<Cc mailing list>
Thanks Olivier. For NVIDIA, the main goal is to have the flexibility to use an out of tree toolchain and remove the hard dependency on the prebuilts folder. Internally, we have pruned the prebuilts to remove dtc, googletest too.
With vacations, we have not verified the latest patches - will test and report in a week.
-----Original Message-----
From: Olivier Deprez <Olivier.Deprez(a)arm.com>
Sent: Monday, 3 January 2022 2:08 PM
To: Bo Yan <byan(a)nvidia.com>; raghu.ncstate(a)icloud.com
Cc: Varun Wadekar <vwadekar(a)nvidia.com>
Subject: Re: [Hafnium] .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Bo, Raghu,
I knew this change would be a bit controversial :) We've had the following main queries:
1/ submodule(s) size is too large (see Varun's very first message in this thread). Starting with prebuilts, it turns out that llvm+gnu toolchains are top contributors. The mentioned change (on top of earlier clang migration changes) introduced the ability to remove a hard dependency to those toolchains.
2/ Ability to build on arm64 host. The change removes the compiler/linker/tools hard coded paths to the x86 prebuilt toolchain from the build system.
3/ toolchain is hardcoded to a fixed version and buried into the project along with sources. This fits developer habits but it is not suitable to a production environment.
Addressing few of the concerns below:
> The default built at the top of tree seems broken. Typing make does not work any more. I tried to export the clang path and I see issues with standard headers...docs/GettingStarted is incomplete?
This is the glitch I expected to happen when pulling latest changes. Hence the reason for the earlier heads up on the ML. Hopefully this recovers after cleaning the tree once, or restarting from a freshly cloned tree.
> I'm not sure I like the idea of removing the prebuilts. We need one tool chain that we know is supported and tested with to create reproducible builds.
ATM the CI still points to the hafnium prebuilt toolchain. Hence builds are still 'reproducible'. It could be proposed that the CI environment itself provides the toolchain.
> We can remove the dependency on the prebuilt in that it should be
> possible for someone to NOT require downloading the upstream tool
> chain if they want to use an alternate chain,
I get the idea, but we have two orthogonal asks (use an external toolchain and remove from prebuilts, or keep in prebuilt but clone conditionally). The .gitmodules does not provide this flexibility AFAIK.
I was thinking of using some partial clone command like below omitting prebuilts:
git submodule update --init driver/linux project/reference third_party/dtc third_party/googletest third_party/linux
(but this doesn't look great from an UX perspective).
> for reference platforms like FVP, the toolchain should be a part of
> the build
Not sure this is a hard dependency. Comparing with TF-A (or other firmware projects), the toolchain is certainly not held within the tree.
The developer or the CI environment provides the toolchain to build the project.
> The current method was ideal but it seems like some latest changes have broken the dev flow...
Ideal for a developer sticking to the provided fixed-version android-based prebuilt toolchain, and building on an x86 host.
Not ideal if one needs to build on arm64, or in a production build when hafnium is considered one component among many others within a (yocto-like) distribution.
> I agree on the point that keeping prebuilts brings benefits, but the build system should be flexible enough such that an out-of-tree toolchain can be easily selected.
This is what this first set of changes provide.
Regards,
Olivier.
________________________________________
From: Bo Yan <byan(a)nvidia.com>
Sent: 21 December 2021 00:45
To: raghu.ncstate(a)icloud.com; Varun Wadekar; Olivier Deprez
Cc: 'Raghu Krishnamurthy via Hafnium'
Subject: RE: [Hafnium] .git submodules increase hafnium code size
I agree on the point that keeping prebuilts brings benefits, but the build system should be flexible enough such that an out-of-tree toolchain can be easily selected.
> -----Original Message-----
> From: raghu.ncstate(a)icloud.com <raghu.ncstate(a)icloud.com>
> Sent: Saturday, December 18, 2021 1:22 PM
> To: raghu.ncstate(a)icloud.com; Varun Wadekar <vwadekar(a)nvidia.com>;
> 'Olivier Deprez' <Olivier.Deprez(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; 'Raghu Krishnamurthy via Hafnium'
> <hafnium(a)lists.trustedfirmware.org>
> Subject: RE: [Hafnium] .git submodules increase hafnium code size
>
> Never mind. Sorry for the false alarm. Had to use make clobber( found
> out once I caught up on the email list...).
> Other comments about continuing to have a prebuilts directory like we
> have today still apply.
>
> -----Original Message-----
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> On Behalf Of
> Raghu Krishnamurthy via Hafnium
> Sent: Saturday, December 18, 2021 1:08 PM
> To: 'Varun Wadekar' <vwadekar(a)nvidia.com>; 'Olivier Deprez'
> <Olivier.Deprez(a)arm.com>
> Cc: 'Bo Yan' <byan(a)nvidia.com>; 'Raghu Krishnamurthy via Hafnium'
> <hafnium(a)lists.trustedfirmware.org>
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> Add hafnium list.
>
> Fix for broken build at -
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Frevi
> e w.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium%2F%2B%2F13150&am
> p;data=04%7C01%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26
> c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63775459320
> 6358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9C%2F0A6
> 1IlusyOnozlGCclZN13YPjaRm1TenBDpemK8M%3D&reserved=0
>
>
> -----Original Message-----
> From: raghu.ncstate(a)icloud.com <raghu.ncstate(a)icloud.com>
> Sent: Saturday, December 18, 2021 12:51 PM
> To: 'Varun Wadekar' <vwadekar(a)nvidia.com>; 'Olivier Deprez'
> <Olivier.Deprez(a)arm.com>
> Cc: 'Bo Yan' <byan(a)nvidia.com>
> Subject: RE: [Hafnium] .git submodules increase hafnium code size
>
> The default built at the top of tree seems broken. Typing make does
> not work any more. I tried to export the clang path and I see issues
> with standard headers...docs/GettingStarted is incomplete?
> I'm not sure I like the idea of removing the prebuilts. We need one
> tool chain that we know is supported and tested with to create reproducible builds.
> We can remove the dependency on the prebuilt in that it should be
> possible for someone to NOT require downloading the upstream tool
> chain if they want to use an alternate chain, but for reference
> platforms like FVP, the toolchain should be a part of the build. The
> current method was ideal but it seems like some latest changes have broken the dev flow...
>
> -----Original Message-----
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> On Behalf Of
> Varun Wadekar via Hafnium
> Sent: Friday, December 17, 2021 7:58 AM
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> Thanks Olivier. Makes sense to me. We are deploying these changes to
> our build system. Will let you know.
>
> -----Original Message-----
> From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Sent: Friday, 17 December 2021 3:32 PM
> To: Varun Wadekar <vwadekar(a)nvidia.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> I don't believe the prebuilt submodule would be fully removed. There
> are libraries and binaries required by the build in any case in the
> linux-aarch64 directory. I rather thought the toolchains within it
> might be removed. They are the main contributor in terms of disk space
> savings which is the problem if I understood properly.
>
> As an intermediate step, before removing the toolchain, I would like
> to confirm that you are able to switch to another toolchain and there
> is no longer a dependency to the toolchain stored in prebuilt.
>
> Regards,
> Olivier.
>
> ________________________________________
> From: Varun Wadekar <vwadekar(a)nvidia.com>
> Sent: 17 December 2021 15:18
> To: Olivier Deprez
> Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
> Subject: RE: .git submodules increase hafnium code size
>
> Thanks Olivier. We hit the toolchain changes yesterday and still
> navigating a course to enable it internally.
>
> When do you plan to remove the prebuilt submodule completely?
>
> -----Original Message-----
> From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Sent: Friday, 17 December 2021 12:00 PM
> To: Varun Wadekar <vwadekar(a)nvidia.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> Just a quick sync on what's been going since this last email.
>
> A number of changes have been made to:
> -remove the gcc dependency, hafnium now only requires clang to build.
> -ability to provide a toolchain different from the one stored in
> prebuilt submodule (merged yesterday) -migrated and tested with recent
> clang versions (was clang 9) -ability to build on arm64 host (our
> need)
>
> The prebuilt submodule still exists but the dependency to hardcoded
> toolchains has weakened.
>
> I intend to tell more about those recent changes and what remains
> during a tech forum early next year, among:
> -remove toolchains from prebuilt?
> -split the hypervisor and spmc test configs into separate project/*
> directories -lessen the dependency to other 3rd party projects
>
> Regards,
> Olivier.
>
> ________________________________________
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of
> Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
> Sent: 08 June 2021 16:00
> To: Varun Wadekar
> Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> Hi Varun,
>
> As indicated earlier, we (Arm side) don't expect to progress on this
> front before Q3.
>
> I'm gathering requirements and expect to discuss through a short
> presentation later in a tech forum or the ML.
>
> Regards,
> Olivier.
>
> ________________________________________
> From: Varun Wadekar <vwadekar(a)nvidia.com>
> Sent: 08 June 2021 15:21
> To: Arunachalam Ganapathy; Olivier Deprez
> Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
> Subject: RE: .git submodules increase hafnium code size
>
> Thanks Arun. We have taken the same approach. The changes I posted
> earlier expect all dependencies to be out of tree and provide
> mechanisms to pass the locations to the make system during compilation.
>
> This is a real problem for us, and we would like a solution that works
> for the community too. @Olivier how should we move forward?
>
> -----Original Message-----
> From: Arunachalam Ganapathy <Arunachalam.Ganapathy(a)arm.com>
> Sent: Tuesday, June 8, 2021 2:08 PM
> To: Varun Wadekar <vwadekar(a)nvidia.com>; Olivier Deprez
> <Olivier.Deprez(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> >> 1- First in context of Total Compute delivery from Arm OSS platforms:
> >> a. ability to build only the SPMC on TC0 platform (not all
> >> reference
> targets such as qemu, rpi4, fvp)
> >> b. use a Yocto provided toolchain.
> >> @Arun, your view on how those two items were solved is
> >> beneficial to
> further elaborate our plans.
>
> For Total Compute we wanted to skip cloning submodules (like
> driver/linux, linux,
> dtc) as only secure_hafnium.bin was required. Basically build only
> reference spm.
> Like: make PROJECT=reference_spm PLAT=TC this builds only secure
> hafnium for one platform.
>
> There were some efforts put on 1.a and was able to build reference_spm
> for one platform. But Hafnium build inside yocto forced to clone all
> submodules (due to dependencies on prebuilt toolchains), so the
> changes were dropped and also the changes weren't clean enough to be upstreamed.
> Regarding 1.b we didn't try hafnium build using yocto toolchain.
>
> Thanks,
> Arun
>
> -----Original Message-----
> From: Varun Wadekar <vwadekar(a)nvidia.com>
> Sent: Monday, June 7, 2021 14:43
> To: Varun Wadekar <vwadekar(a)nvidia.com>; Olivier Deprez
> <Olivier.Deprez(a)arm.com>; Arunachalam Ganapathy
> <Arunachalam.Ganapathy(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> <hafnium(a)lists.trustedfirmware.org>
> Subject: RE: .git submodules increase hafnium code size
>
> Hi,
>
> >> @Arun, your view on how those two items were solved is beneficial
> >> to
> further elaborate our plans.
>
> @Arunachalam Ganapathy your comments on this topic would be very
> helpful.
>
> Thanks.
>
> -----Original Message-----
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> On Behalf Of
> Varun Wadekar via Hafnium
> Sent: Monday, May 31, 2021 1:49 PM
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>;
> hafnium(a)lists.trustedfirmware.org; Arunachalam Ganapathy
> <Arunachalam.Ganapathy(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Olivier,
>
> Thanks for answering my queries.
>
> We are looking to deploy the following use case at NVIDIA.
>
> <snip>
>
> -ability to build only the SPMC (not all reference targets such as
> qemu, rpi4,
> fvp) -A distribution only requiring the Hypervisor/SPMC output binary
> ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or
> x86 host, and arbitrary clang version).
>
> <snip>
>
> >> As you noticed, the Hafnium Hypervisor/SPMC and test environment
> builds are closely coupled by the use of ninja/gn flow and scripts. We
> intend to approach those problems in the course of Q3 in Arm OSS roadmap.
>
> [VW] Are there any local changes to decouple hafnium from its dependencies?
> We can evaluate Arm;s approach against what we use internally. Our
> changes moved the dependencies out of the tree and passed file
> locations to the build system with the help of command line arguments.
>
> -Varun
>
> -----Original Message-----
> From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Sent: Monday, May 31, 2021 11:03 AM
> To: hafnium(a)lists.trustedfirmware.org; Varun Wadekar
> <vwadekar(a)nvidia.com>; Arunachalam Ganapathy
> <Arunachalam.Ganapathy(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> We had similar requests raised internally.
> 1- First in context of Total Compute delivery from Arm OSS platforms:
> a. ability to build only the SPMC on TC0 platform (not all
> reference targets such as qemu, rpi4, fvp)
> b. use a Yocto provided toolchain.
> @Arun, your view on how those two items were solved is beneficial
> to further elaborate our plans.
> 2- A similar request as 1.b to build Hafnium as part of a distribution
> on
> arm64 host:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeve
> loper.trustedfirmware.org%2FT898&data=04%7C01%7Cbyan%40nvidia.
> com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637754593206358976%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> 0%3D%7C3000&sdata=AtMec%2BNFfhA14xeMZT992icEj6SLAhwLnB2co5
> VXFNM%3D&reserved=0
>
> In my view there are two consumers:
> -A distribution only requiring the Hypervisor/SPMC output binary
> ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or
> x86 host, and arbitrary clang version).
> -The Hf CI framework/automation needs the above, plus the test
> framework and tests (dependency on googletest, linux submodules etc).
> It's important to keep this item alive while trying to solve above item.
>
> As you noticed, the Hafnium Hypervisor/SPMC and test environment
> builds are closely coupled by the use of ninja/gn flow and scripts.
> They are using a fixed toolchain version through prebuilts to ensure
> builds are "reproducible", in particular with regards to the Hafnium CI.
>
> We intend to approach those problems in the course of Q3 in Arm OSS
> roadmap.
> As an early exploration we already have:
> -clang 12 compiler upgrade. This is necessary if wiling to use any
> arbitrary clang version:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Frevi
> e
> w.trustedfirmware.org%2Fq%2Ftopic%3A%2522od%25252Fhf-
> clang12%2522&data=04%7C01%7Cbyan%40nvidia.com%7C030b52fc550
> 94b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C
> 0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL
> jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> ;sdata=t7KJjvv2o8EHKBpPdUEIrJ%2F7Vy7KFEODdnQMrL4LpaU%3D&res
> erved=0+(status:open%20OR%20status:merged)
> -Ability to build on arm64 host (done, internally).
> -Identify the flow/script changes such that external dependencies can
> be used (on-going, internally).
> I thought of localizing common dependencies to python/shell scripts by
> the use of definition files included in the mentioned scripts. This is
> only an early investigation, I will check how this intersects the changes you provided.
>
> Regards,
> Olivier.
>
>
>
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of
> Varun Wadekar via Hafnium <hafnium(a)lists.trustedfirmware.org>
> Sent: 28 May 2021 16:47
> To: hafnium(a)lists.trustedfirmware.org
> <hafnium(a)lists.trustedfirmware.org>
> Cc: Bo Yan <byan(a)nvidia.com>
> Subject: [Hafnium] .git submodules increase hafnium code size
>
> Hi,
>
> We at NVIDIA are evaluating Hafnium. During the initial investigation,
> we found out that the repository size (in terms of MB) is huge. This
> is mostly because of the "git submodules" used by the project. This is
> a great way to deliver Hafnium with its dependencies in one go.
>
> But we think that the size can be trimmed by moving the toolchain,
> linux folder, googletest and dtc compiler out, leaving just the
> Hafnium code in the project. This way, companies like us can pick and
> choose instead of having to use everything. In a bid to ease the pain
> internally and only use the Hafnium code base we have crafted the following changes:
>
>
> 1. hafnium: support external projects (I10a07de3) * Gerrit Code
> Review
> (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?
> url =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium
> %2F%2B%2F10142&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f
> c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0
> %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> amp;sdata=7PWBcTXxP6HtqN%2BTGz3cEnnfabSzlyjo0RNZOjvOL4A%3D&
> ;reserved=0>
> 2. hafnium: build with dtc and googletest out of tree (I057c9ad6) *
> Gerrit Code Review
> (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?
> url =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium
> %2F%2B%2F10144&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f
> c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0
> %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> amp;sdata=Ql8XW8FAEQ%2FereVjB3glccWJLR7sQ3yg0v9%2FSptCnyk%3D&a
> mp;reserved=0>
> 3. build: support external toolchain (Iafd029c1) * Gerrit Code
> Review
> (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?
> url =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium
> %2F%2B%2F10145&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f
> c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0
> %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> amp;sdata=X%2FNE1WmrcQvqRzEBq8UXdA02y9rl%2BS%2Fs1btuRAAlacM%3
> D&reserved=0>
>
> This series does not have the patch to use an out of tree linux
> codebase. I assume these patches wont be acceptable in their current
> state, so would like to know how the community plans to handle this situation.
>
> The code size is a real concern for us, as we already have copies of
> the dependencies in our codebase, so have no use for these duplicates.
>
> Thanks.
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.t%2F&data=04%7C01%7Cvwadekar%40nvidia.com%7Cb9b5a4a5f7034f3e68ac
> 08d9cec2749e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637768156850
> 567117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Szglo5hCctZ6EMQIgkKd0Ya0F
> aygD4b77ubvxnnOFyg%3D&reserved=0
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.t%2F&data=04%7C01%7Cvwadekar%40nvidia.com%7Cb9b5a4a5f7034f3e68ac
> 08d9cec2749e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637768156850
> 567117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Szglo5hCctZ6EMQIgkKd0Ya0F
> aygD4b77ubvxnnOFyg%3D&reserved=0
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.t%2F&data=04%7C01%7Cvwadekar%40nvidia.com%7Cb9b5a4a5f7034f3e68ac
> 08d9cec2749e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637768156850
> 567117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Szglo5hCctZ6EMQIgkKd0Ya0F
> aygD4b77ubvxnnOFyg%3D&reserved=0
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.t%2F&data=04%7C01%7Cvwadekar%40nvidia.com%7Cb9b5a4a5f7034f3e68ac
> 08d9cec2749e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637768156850
> 567117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Szglo5hCctZ6EMQIgkKd0Ya0F
> aygD4b77ubvxnnOFyg%3D&reserved=0
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.t%2F&data=04%7C01%7Cvwadekar%40nvidia.com%7Cb9b5a4a5f7034f3e68ac
> 08d9cec2749e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637768156850
> 567117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Szglo5hCctZ6EMQIgkKd0Ya0F
> aygD4b77ubvxnnOFyg%3D&reserved=0
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi,
Can someone help me figure this out? I am trying to send a message from my
Linux primary VM to a secondary VM, but for some reason ffa_msg_send keeps
on failing and I keep getting an EAGAIN error: resource temporarily
unavailable.
Any suggestions?
Best,
Friedrich
+ ML
________________________________________
From: Olivier Deprez <Olivier.Deprez(a)arm.com>
Sent: 03 January 2022 15:07
To: Bo Yan; raghu.ncstate(a)icloud.com
Cc: Varun Wadekar
Subject: Re: [Hafnium] .git submodules increase hafnium code size
Hi Bo, Raghu,
I knew this change would be a bit controversial :)
We've had the following main queries:
1/ submodule(s) size is too large (see Varun's very first message in this thread). Starting with prebuilts, it turns out that llvm+gnu toolchains are top contributors. The mentioned change (on top of earlier clang migration changes) introduced the ability to remove a hard dependency to those toolchains.
2/ Ability to build on arm64 host. The change removes the compiler/linker/tools hard coded paths to the x86 prebuilt toolchain from the build system.
3/ toolchain is hardcoded to a fixed version and buried into the project along with sources. This fits developer habits but it is not suitable to a production environment.
Addressing few of the concerns below:
> The default built at the top of tree seems broken. Typing make does not work any more. I tried to export the clang path and I see issues with standard headers...docs/GettingStarted is incomplete?
This is the glitch I expected to happen when pulling latest changes. Hence the reason for the earlier heads up on the ML. Hopefully this recovers after cleaning the tree once, or restarting from a freshly cloned tree.
> I'm not sure I like the idea of removing the prebuilts. We need one tool chain that we know is supported and tested with to create reproducible builds.
ATM the CI still points to the hafnium prebuilt toolchain. Hence builds are still 'reproducible'. It could be proposed that the CI environment itself provides the toolchain.
> We can remove the dependency on the prebuilt in that it should be possible for someone to NOT require downloading the upstream tool chain if they want to use an alternate chain,
I get the idea, but we have two orthogonal asks (use an external toolchain and remove from prebuilts, or keep in prebuilt but clone conditionally). The .gitmodules does not provide this flexibility AFAIK.
I was thinking of using some partial clone command like below omitting prebuilts:
git submodule update --init driver/linux project/reference third_party/dtc third_party/googletest third_party/linux
(but this doesn't look great from an UX perspective).
> for reference platforms like FVP, the toolchain should be a part of the build
Not sure this is a hard dependency. Comparing with TF-A (or other firmware projects), the toolchain is certainly not held within the tree.
The developer or the CI environment provides the toolchain to build the project.
> The current method was ideal but it seems like some latest changes have broken the dev flow...
Ideal for a developer sticking to the provided fixed-version android-based prebuilt toolchain, and building on an x86 host.
Not ideal if one needs to build on arm64, or in a production build when hafnium is considered one component among many others within a (yocto-like) distribution.
> I agree on the point that keeping prebuilts brings benefits, but the build system should be flexible enough such that an out-of-tree toolchain can be easily selected.
This is what this first set of changes provide.
Regards,
Olivier.
________________________________________
From: Bo Yan <byan(a)nvidia.com>
Sent: 21 December 2021 00:45
To: raghu.ncstate(a)icloud.com; Varun Wadekar; Olivier Deprez
Cc: 'Raghu Krishnamurthy via Hafnium'
Subject: RE: [Hafnium] .git submodules increase hafnium code size
I agree on the point that keeping prebuilts brings benefits, but the build system should be flexible enough such that an out-of-tree toolchain can be easily selected.
> -----Original Message-----
> From: raghu.ncstate(a)icloud.com <raghu.ncstate(a)icloud.com>
> Sent: Saturday, December 18, 2021 1:22 PM
> To: raghu.ncstate(a)icloud.com; Varun Wadekar <vwadekar(a)nvidia.com>;
> 'Olivier Deprez' <Olivier.Deprez(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; 'Raghu Krishnamurthy via Hafnium'
> <hafnium(a)lists.trustedfirmware.org>
> Subject: RE: [Hafnium] .git submodules increase hafnium code size
>
> Never mind. Sorry for the false alarm. Had to use make clobber( found out
> once I caught up on the email list...).
> Other comments about continuing to have a prebuilts directory like we have
> today still apply.
>
> -----Original Message-----
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> On Behalf Of
> Raghu Krishnamurthy via Hafnium
> Sent: Saturday, December 18, 2021 1:08 PM
> To: 'Varun Wadekar' <vwadekar(a)nvidia.com>; 'Olivier Deprez'
> <Olivier.Deprez(a)arm.com>
> Cc: 'Bo Yan' <byan(a)nvidia.com>; 'Raghu Krishnamurthy via Hafnium'
> <hafnium(a)lists.trustedfirmware.org>
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> Add hafnium list.
>
> Fix for broken build at -
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Frevie
> w.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium%2F%2B%2F13150&am
> p;data=04%7C01%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26
> c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63775459320
> 6358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9C%2F0A6
> 1IlusyOnozlGCclZN13YPjaRm1TenBDpemK8M%3D&reserved=0
>
>
> -----Original Message-----
> From: raghu.ncstate(a)icloud.com <raghu.ncstate(a)icloud.com>
> Sent: Saturday, December 18, 2021 12:51 PM
> To: 'Varun Wadekar' <vwadekar(a)nvidia.com>; 'Olivier Deprez'
> <Olivier.Deprez(a)arm.com>
> Cc: 'Bo Yan' <byan(a)nvidia.com>
> Subject: RE: [Hafnium] .git submodules increase hafnium code size
>
> The default built at the top of tree seems broken. Typing make does not work
> any more. I tried to export the clang path and I see issues with standard
> headers...docs/GettingStarted is incomplete?
> I'm not sure I like the idea of removing the prebuilts. We need one tool chain
> that we know is supported and tested with to create reproducible builds.
> We can remove the dependency on the prebuilt in that it should be possible
> for someone to NOT require downloading the upstream tool chain if they
> want to use an alternate chain, but for reference platforms like FVP, the
> toolchain should be a part of the build. The current method was ideal but it
> seems like some latest changes have broken the dev flow...
>
> -----Original Message-----
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> On Behalf Of
> Varun Wadekar via Hafnium
> Sent: Friday, December 17, 2021 7:58 AM
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> Thanks Olivier. Makes sense to me. We are deploying these changes to our
> build system. Will let you know.
>
> -----Original Message-----
> From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Sent: Friday, 17 December 2021 3:32 PM
> To: Varun Wadekar <vwadekar(a)nvidia.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> I don't believe the prebuilt submodule would be fully removed. There are
> libraries and binaries required by the build in any case in the linux-aarch64
> directory. I rather thought the toolchains within it might be removed. They
> are the main contributor in terms of disk space savings which is the problem
> if I understood properly.
>
> As an intermediate step, before removing the toolchain, I would like to
> confirm that you are able to switch to another toolchain and there is no
> longer a dependency to the toolchain stored in prebuilt.
>
> Regards,
> Olivier.
>
> ________________________________________
> From: Varun Wadekar <vwadekar(a)nvidia.com>
> Sent: 17 December 2021 15:18
> To: Olivier Deprez
> Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
> Subject: RE: .git submodules increase hafnium code size
>
> Thanks Olivier. We hit the toolchain changes yesterday and still navigating a
> course to enable it internally.
>
> When do you plan to remove the prebuilt submodule completely?
>
> -----Original Message-----
> From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Sent: Friday, 17 December 2021 12:00 PM
> To: Varun Wadekar <vwadekar(a)nvidia.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> Just a quick sync on what's been going since this last email.
>
> A number of changes have been made to:
> -remove the gcc dependency, hafnium now only requires clang to build.
> -ability to provide a toolchain different from the one stored in prebuilt
> submodule (merged yesterday) -migrated and tested with recent clang
> versions (was clang 9) -ability to build on arm64 host (our need)
>
> The prebuilt submodule still exists but the dependency to hardcoded
> toolchains has weakened.
>
> I intend to tell more about those recent changes and what remains during a
> tech forum early next year, among:
> -remove toolchains from prebuilt?
> -split the hypervisor and spmc test configs into separate project/* directories
> -lessen the dependency to other 3rd party projects
>
> Regards,
> Olivier.
>
> ________________________________________
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of
> Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
> Sent: 08 June 2021 16:00
> To: Varun Wadekar
> Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> Hi Varun,
>
> As indicated earlier, we (Arm side) don't expect to progress on this front
> before Q3.
>
> I'm gathering requirements and expect to discuss through a short
> presentation later in a tech forum or the ML.
>
> Regards,
> Olivier.
>
> ________________________________________
> From: Varun Wadekar <vwadekar(a)nvidia.com>
> Sent: 08 June 2021 15:21
> To: Arunachalam Ganapathy; Olivier Deprez
> Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
> Subject: RE: .git submodules increase hafnium code size
>
> Thanks Arun. We have taken the same approach. The changes I posted
> earlier expect all dependencies to be out of tree and provide mechanisms to
> pass the locations to the make system during compilation.
>
> This is a real problem for us, and we would like a solution that works for the
> community too. @Olivier how should we move forward?
>
> -----Original Message-----
> From: Arunachalam Ganapathy <Arunachalam.Ganapathy(a)arm.com>
> Sent: Tuesday, June 8, 2021 2:08 PM
> To: Varun Wadekar <vwadekar(a)nvidia.com>; Olivier Deprez
> <Olivier.Deprez(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> >> 1- First in context of Total Compute delivery from Arm OSS platforms:
> >> a. ability to build only the SPMC on TC0 platform (not all reference
> targets such as qemu, rpi4, fvp)
> >> b. use a Yocto provided toolchain.
> >> @Arun, your view on how those two items were solved is beneficial to
> further elaborate our plans.
>
> For Total Compute we wanted to skip cloning submodules (like driver/linux,
> linux,
> dtc) as only secure_hafnium.bin was required. Basically build only reference
> spm.
> Like: make PROJECT=reference_spm PLAT=TC this builds only secure hafnium
> for one platform.
>
> There were some efforts put on 1.a and was able to build reference_spm for
> one platform. But Hafnium build inside yocto forced to clone all submodules
> (due to dependencies on prebuilt toolchains), so the changes were dropped
> and also the changes weren't clean enough to be upstreamed.
> Regarding 1.b we didn't try hafnium build using yocto toolchain.
>
> Thanks,
> Arun
>
> -----Original Message-----
> From: Varun Wadekar <vwadekar(a)nvidia.com>
> Sent: Monday, June 7, 2021 14:43
> To: Varun Wadekar <vwadekar(a)nvidia.com>; Olivier Deprez
> <Olivier.Deprez(a)arm.com>; Arunachalam Ganapathy
> <Arunachalam.Ganapathy(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
> <hafnium(a)lists.trustedfirmware.org>
> Subject: RE: .git submodules increase hafnium code size
>
> Hi,
>
> >> @Arun, your view on how those two items were solved is beneficial to
> further elaborate our plans.
>
> @Arunachalam Ganapathy your comments on this topic would be very
> helpful.
>
> Thanks.
>
> -----Original Message-----
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> On Behalf Of
> Varun Wadekar via Hafnium
> Sent: Monday, May 31, 2021 1:49 PM
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>;
> hafnium(a)lists.trustedfirmware.org; Arunachalam Ganapathy
> <Arunachalam.Ganapathy(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>
> Subject: Re: [Hafnium] .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Olivier,
>
> Thanks for answering my queries.
>
> We are looking to deploy the following use case at NVIDIA.
>
> <snip>
>
> -ability to build only the SPMC (not all reference targets such as qemu, rpi4,
> fvp) -A distribution only requiring the Hypervisor/SPMC output binary
> ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or x86
> host, and arbitrary clang version).
>
> <snip>
>
> >> As you noticed, the Hafnium Hypervisor/SPMC and test environment
> builds are closely coupled by the use of ninja/gn flow and scripts. We intend
> to approach those problems in the course of Q3 in Arm OSS roadmap.
>
> [VW] Are there any local changes to decouple hafnium from its dependencies?
> We can evaluate Arm;s approach against what we use internally. Our
> changes moved the dependencies out of the tree and passed file locations to
> the build system with the help of command line arguments.
>
> -Varun
>
> -----Original Message-----
> From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Sent: Monday, May 31, 2021 11:03 AM
> To: hafnium(a)lists.trustedfirmware.org; Varun Wadekar
> <vwadekar(a)nvidia.com>; Arunachalam Ganapathy
> <Arunachalam.Ganapathy(a)arm.com>
> Cc: Bo Yan <byan(a)nvidia.com>
> Subject: Re: .git submodules increase hafnium code size
>
> External email: Use caution opening links or attachments
>
>
> Hi Varun,
>
> We had similar requests raised internally.
> 1- First in context of Total Compute delivery from Arm OSS platforms:
> a. ability to build only the SPMC on TC0 platform (not all reference targets
> such as qemu, rpi4, fvp)
> b. use a Yocto provided toolchain.
> @Arun, your view on how those two items were solved is beneficial to
> further elaborate our plans.
> 2- A similar request as 1.b to build Hafnium as part of a distribution on
> arm64 host:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeve
> loper.trustedfirmware.org%2FT898&data=04%7C01%7Cbyan%40nvidia.
> com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637754593206358976%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> 0%3D%7C3000&sdata=AtMec%2BNFfhA14xeMZT992icEj6SLAhwLnB2co5
> VXFNM%3D&reserved=0
>
> In my view there are two consumers:
> -A distribution only requiring the Hypervisor/SPMC output binary
> ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or x86
> host, and arbitrary clang version).
> -The Hf CI framework/automation needs the above, plus the test framework
> and tests (dependency on googletest, linux submodules etc). It's important to
> keep this item alive while trying to solve above item.
>
> As you noticed, the Hafnium Hypervisor/SPMC and test environment builds
> are closely coupled by the use of ninja/gn flow and scripts.
> They are using a fixed toolchain version through prebuilts to ensure builds
> are "reproducible", in particular with regards to the Hafnium CI.
>
> We intend to approach those problems in the course of Q3 in Arm OSS
> roadmap.
> As an early exploration we already have:
> -clang 12 compiler upgrade. This is necessary if wiling to use any arbitrary
> clang version:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Frevie
> w.trustedfirmware.org%2Fq%2Ftopic%3A%2522od%25252Fhf-
> clang12%2522&data=04%7C01%7Cbyan%40nvidia.com%7C030b52fc550
> 94b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C
> 0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL
> jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> ;sdata=t7KJjvv2o8EHKBpPdUEIrJ%2F7Vy7KFEODdnQMrL4LpaU%3D&res
> erved=0+(status:open%20OR%20status:merged)
> -Ability to build on arm64 host (done, internally).
> -Identify the flow/script changes such that external dependencies can be
> used (on-going, internally).
> I thought of localizing common dependencies to python/shell scripts by the
> use of definition files included in the mentioned scripts. This is only an early
> investigation, I will check how this intersects the changes you provided.
>
> Regards,
> Olivier.
>
>
>
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of
> Varun Wadekar via Hafnium <hafnium(a)lists.trustedfirmware.org>
> Sent: 28 May 2021 16:47
> To: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
> Cc: Bo Yan <byan(a)nvidia.com>
> Subject: [Hafnium] .git submodules increase hafnium code size
>
> Hi,
>
> We at NVIDIA are evaluating Hafnium. During the initial investigation, we
> found out that the repository size (in terms of MB) is huge. This is mostly
> because of the "git submodules" used by the project. This is a great way to
> deliver Hafnium with its dependencies in one go.
>
> But we think that the size can be trimmed by moving the toolchain, linux
> folder, googletest and dtc compiler out, leaving just the Hafnium code in the
> project. This way, companies like us can pick and choose instead of having to
> use everything. In a bid to ease the pain internally and only use the Hafnium
> code base we have crafted the following changes:
>
>
> 1. hafnium: support external projects (I10a07de3) * Gerrit Code Review
> (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?url
> =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium
> %2F%2B%2F10142&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f
> c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0
> %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> amp;sdata=7PWBcTXxP6HtqN%2BTGz3cEnnfabSzlyjo0RNZOjvOL4A%3D&
> ;reserved=0>
> 2. hafnium: build with dtc and googletest out of tree (I057c9ad6) * Gerrit
> Code Review
> (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?url
> =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium
> %2F%2B%2F10144&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f
> c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0
> %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> amp;sdata=Ql8XW8FAEQ%2FereVjB3glccWJLR7sQ3yg0v9%2FSptCnyk%3D&a
> mp;reserved=0>
> 3. build: support external toolchain (Iafd029c1) * Gerrit Code Review
> (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?url
> =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium
> %2F%2B%2F10145&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f
> c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0
> %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> amp;sdata=X%2FNE1WmrcQvqRzEBq8UXdA02y9rl%2BS%2Fs1btuRAAlacM%3
> D&reserved=0>
>
> This series does not have the patch to use an out of tree linux codebase. I
> assume these patches wont be acceptable in their current state, so would
> like to know how the community plans to handle this situation.
>
> The code size is a real concern for us, as we already have copies of the
> dependencies in our codebase, so have no use for these duplicates.
>
> Thanks.
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t
> rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0
> 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308
> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y
> ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
Hi all,
When data or instruction abort happens, Hafnium provides the esr_el2,
far, pc values and etc. I think this info is useful, but I want to
know how to debug with the PC value.
I know TF-A provides some dump files (e.g., bl31.dump). So, when some
panic or errors happen, I can locate the "bug instruction", its
functions, and other useful context quickly.
Will Hafnium provides similar dump files? If not, can we create them? How?
Sincerely,
Wang
Hi Varun,
Just a quick sync on what's been going since this last email.
A number of changes have been made to:
-remove the gcc dependency, hafnium now only requires clang to build.
-ability to provide a toolchain different from the one stored in prebuilt submodule (merged yesterday)
-migrated and tested with recent clang versions (was clang 9)
-ability to build on arm64 host (our need)
The prebuilt submodule still exists but the dependency to hardcoded toolchains has weakened.
I intend to tell more about those recent changes and what remains during a tech forum early next year, among:
-remove toolchains from prebuilt?
-split the hypervisor and spmc test configs into separate project/* directories
-lessen the dependency to other 3rd party projects
Regards,
Olivier.
________________________________________
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 08 June 2021 16:00
To: Varun Wadekar
Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
Subject: Re: [Hafnium] .git submodules increase hafnium code size
Hi Varun,
As indicated earlier, we (Arm side) don't expect to progress on this front before Q3.
I'm gathering requirements and expect to discuss through a short presentation later in a tech forum or the ML.
Regards,
Olivier.
________________________________________
From: Varun Wadekar <vwadekar(a)nvidia.com>
Sent: 08 June 2021 15:21
To: Arunachalam Ganapathy; Olivier Deprez
Cc: Bo Yan; hafnium(a)lists.trustedfirmware.org
Subject: RE: .git submodules increase hafnium code size
Thanks Arun. We have taken the same approach. The changes I posted earlier expect all dependencies to be out of tree and provide mechanisms to pass the locations to the make system during compilation.
This is a real problem for us, and we would like a solution that works for the community too. @Olivier how should we move forward?
-----Original Message-----
From: Arunachalam Ganapathy <Arunachalam.Ganapathy(a)arm.com>
Sent: Tuesday, June 8, 2021 2:08 PM
To: Varun Wadekar <vwadekar(a)nvidia.com>; Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org
Subject: Re: .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Varun,
>> 1- First in context of Total Compute delivery from Arm OSS platforms:
>> a. ability to build only the SPMC on TC0 platform (not all reference targets such as qemu, rpi4, fvp)
>> b. use a Yocto provided toolchain.
>> @Arun, your view on how those two items were solved is beneficial to further elaborate our plans.
For Total Compute we wanted to skip cloning submodules (like driver/linux, linux,
dtc) as only secure_hafnium.bin was required. Basically build only reference spm.
Like: make PROJECT=reference_spm PLAT=TC this builds only secure hafnium for one platform.
There were some efforts put on 1.a and was able to build reference_spm for one platform. But Hafnium build inside yocto forced to clone all submodules (due to dependencies on prebuilt toolchains), so the changes were dropped and also the changes weren't clean enough to be upstreamed.
Regarding 1.b we didn't try hafnium build using yocto toolchain.
Thanks,
Arun
-----Original Message-----
From: Varun Wadekar <vwadekar(a)nvidia.com>
Sent: Monday, June 7, 2021 14:43
To: Varun Wadekar <vwadekar(a)nvidia.com>; Olivier Deprez <Olivier.Deprez(a)arm.com>; Arunachalam Ganapathy <Arunachalam.Ganapathy(a)arm.com>
Cc: Bo Yan <byan(a)nvidia.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
Subject: RE: .git submodules increase hafnium code size
Hi,
>> @Arun, your view on how those two items were solved is beneficial to further elaborate our plans.
@Arunachalam Ganapathy your comments on this topic would be very helpful.
Thanks.
-----Original Message-----
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> On Behalf Of Varun Wadekar via Hafnium
Sent: Monday, May 31, 2021 1:49 PM
To: Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium(a)lists.trustedfirmware.org; Arunachalam Ganapathy <Arunachalam.Ganapathy(a)arm.com>
Cc: Bo Yan <byan(a)nvidia.com>
Subject: Re: [Hafnium] .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Olivier,
Thanks for answering my queries.
We are looking to deploy the following use case at NVIDIA.
<snip>
-ability to build only the SPMC (not all reference targets such as qemu, rpi4, fvp) -A distribution only requiring the Hypervisor/SPMC output binary ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or x86 host, and arbitrary clang version).
<snip>
>> As you noticed, the Hafnium Hypervisor/SPMC and test environment builds are closely coupled by the use of ninja/gn flow and scripts. We intend to approach those problems in the course of Q3 in Arm OSS roadmap.
[VW] Are there any local changes to decouple hafnium from its dependencies? We can evaluate Arm;s approach against what we use internally. Our changes moved the dependencies out of the tree and passed file locations to the build system with the help of command line arguments.
-Varun
-----Original Message-----
From: Olivier Deprez <Olivier.Deprez(a)arm.com>
Sent: Monday, May 31, 2021 11:03 AM
To: hafnium(a)lists.trustedfirmware.org; Varun Wadekar <vwadekar(a)nvidia.com>; Arunachalam Ganapathy <Arunachalam.Ganapathy(a)arm.com>
Cc: Bo Yan <byan(a)nvidia.com>
Subject: Re: .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Varun,
We had similar requests raised internally.
1- First in context of Total Compute delivery from Arm OSS platforms:
a. ability to build only the SPMC on TC0 platform (not all reference targets such as qemu, rpi4, fvp)
b. use a Yocto provided toolchain.
@Arun, your view on how those two items were solved is beneficial to further elaborate our plans.
2- A similar request as 1.b to build Hafnium as part of a distribution on arm64 host: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper…
In my view there are two consumers:
-A distribution only requiring the Hypervisor/SPMC output binary ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or x86 host, and arbitrary clang version).
-The Hf CI framework/automation needs the above, plus the test framework and tests (dependency on googletest, linux submodules etc). It's important to keep this item alive while trying to solve above item.
As you noticed, the Hafnium Hypervisor/SPMC and test environment builds are closely coupled by the use of ninja/gn flow and scripts.
They are using a fixed toolchain version through prebuilts to ensure builds are "reproducible", in particular with regards to the Hafnium CI.
We intend to approach those problems in the course of Q3 in Arm OSS roadmap.
As an early exploration we already have:
-clang 12 compiler upgrade. This is necessary if wiling to use any arbitrary clang version:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.tr…
-Ability to build on arm64 host (done, internally).
-Identify the flow/script changes such that external dependencies can be used (on-going, internally).
I thought of localizing common dependencies to python/shell scripts by the use of definition files included in the mentioned scripts. This is only an early investigation, I will check how this intersects the changes you provided.
Regards,
Olivier.
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Varun Wadekar via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 28 May 2021 16:47
To: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
Cc: Bo Yan <byan(a)nvidia.com>
Subject: [Hafnium] .git submodules increase hafnium code size
Hi,
We at NVIDIA are evaluating Hafnium. During the initial investigation, we found out that the repository size (in terms of MB) is huge. This is mostly because of the "git submodules" used by the project. This is a great way to deliver Hafnium with its dependencies in one go.
But we think that the size can be trimmed by moving the toolchain, linux folder, googletest and dtc compiler out, leaving just the Hafnium code in the project. This way, companies like us can pick and choose instead of having to use everything. In a bid to ease the pain internally and only use the Hafnium code base we have crafted the following changes:
1. hafnium: support external projects (I10a07de3) * Gerrit Code Review (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.tr…>
2. hafnium: build with dtc and googletest out of tree (I057c9ad6) * Gerrit Code Review (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.tr…>
3. build: support external toolchain (Iafd029c1) * Gerrit Code Review (trustedfirmware.org)<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.tr…>
This series does not have the patch to use an out of tree linux codebase. I assume these patches wont be acceptable in their current state, so would like to know how the community plans to handle this situation.
The code size is a real concern for us, as we already have copies of the dependencies in our codebase, so have no use for these duplicates.
Thanks.
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.tru…
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.tru…
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
Hi Raghu,
a fix for hftest.py has been merged:
https://review.trustedfirmware.org/c/hafnium/hafnium/+/12481
addressing the random failures of both-world tests and supporting
connection to telnet ports other than 5000
Cheers,
Federico
____________________________________
> From: raghu.ncstate(a)icloud.com <raghu.ncstate(a)icloud.com>
> Sent: 04 August 2021 21:01
> To: Olivier Deprez; 'Raghu Krishnamurthy via Hafnium'
> Subject: RE: [Hafnium] Bug in hftest.py
>
> Thanks Olivier. I've created https://developer.trustedfirmware.org/T955 to
> track. Understood all of this is new. I do have local fixes to get around
> the issue so not a hurry to have a fix merged, but something to consider and
> fix since it will eventually show up.
>
>>> the both worlds test scenario is not 100% stable on my machine
> [RK] Likewise. I've noticed that this is caused by lingering FVP processes.
> Usually I ps -ax | grep for FVP instances, kill and then run tests and I
> never see failures after that. The issue that I faced was that the lingering
> FVP would take up telnet ports and the newly spawned ones use different
> ports(>5004) than what hftest.py expects. It appears that when tests fail,
> we may not be cleaning up/exiting processes properly, but I haven't checked.
> Or the code may be just fine and a ctrl+c leaves those processes lingering.
>
> Thanks
> Raghu
>
> -----Original Message-----
> From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Sent: Tuesday, August 3, 2021 11:51 PM
> To: 'Raghu Krishnamurthy via Hafnium' <hafnium(a)lists.trustedfirmware.org>;
> raghu.ncstate(a)icloud.com
> Subject: Re: [Hafnium] Bug in hftest.py
>
> Hi Raghu,
>
> Thanks for reporting.
> This part of the test infrastructure (testing the SPMC) is still very fresh
> and requires improvement iterations so please bear with us. Also a reason
> it's not yet part of the automated non-regression with jenkins (as opposed
> to the legacy kokoro/test.sh). For the time being we still mostly rely on
> the TF-A CI for testing on the secure side.
>
> IIUC this change was made to help with the test time as the FVP takes long
> to reload on every test.
> But indeed it might have the side effect you describe.
> So either we revert the FVP reloading on every test.
> Or another (somewhat hackish) possibility is to clear the mentioned
> variables from within the test (or make them part of BSS)?
>
> To be fair, the both worlds test scenario is not 100% stable on my machine
> (for some reason the connection is not always successful between the FVP and
> hftest) hence limiting confidence/robustness of my testing and
> investigations. So I wonder is the scripting is still somewhat a bit
> fragile.
>
> Regards,
> Olivier.
>
> ________________________________________
> From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Raghu
> Krishnamurthy via Hafnium <hafnium(a)lists.trustedfirmware.org>
> Sent: 03 August 2021 23:47
> To: 'Raghu Krishnamurthy via Hafnium'
> Subject: [Hafnium] Bug in hftest.py
>
> Hi All,
>
>
>
> Wanted to report to you that commit 18a25f9241f86ba2d637011ff465ce3869e8651b
> in hafnium "appears" broken. The issue with the optimization in this patch
> is that the partition images are not reloaded for each test run, which means
> a previous test could have written data to say SRAM, and the following test
> would use the old values from the previous test, when the same image is
> executed again from SRAM for a following test. This would be a problem for
> pretty much anything in the data section of a partition. In my case, I have
> a counter in the data section of my partition, which does not get reset back
> to its original value.
>
> I've attached a patch to help repro the issue. Fix is to disable the
> optimization or somehow reload the images for each run. This affects only
> "both world" tests.
>
> Let me know if I'm missing something here.
>
>
>
> Apply patch and run timeout --foreground 300s ./test/hftest/hftest.py
> --out_partitions out/reference/secure_aem_v8a_fvp_vm_clang --log
> out/reference/kokoro_log --spmc
> out/reference/secure_aem_v8a_fvp_clang/hafnium.bin --driver=fvp --hypervisor
> out/reference/aem_v8a_fvp_clang/hafnium.bin --partitions_json
> test/vmapi/ffa_secure_partitions/ffa_both_world_partitions_test.json
>
>
>
> The command line is from kokoro/test_spmc.sh.
>
>
>
> Thanks
>
> Raghu
>
>
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
Hi,
A small heads up that from this change: https://review.trustedfirmware.org/c/hafnium/hafnium/+/11613
a developer needs to provide the LLVM/clang toolchain through the PATH environment variable.
See the documentation update:
https://review.trustedfirmware.org/plugins/gitiles/hafnium/hafnium/+/HEAD/d…
Until then the toolchain was hardcoded to use the version stored in the prebuilt submodule.
>From now, this weakens the dependency to the prebuilt toolchain and provides flexibility with providing an alternate toolchain on a x86 host. This also opens to building on an aarch64 host.
This has been tested with different combinations of hosts, ubuntu releases and toolchains downloaded from https://releases.llvm.org/download.html
x86_64 Ubuntu 18.04 / 20.04 clang/llvm 12.0.0 , 13.0.0
aarch64 Ubuntu 19.04 (Ampere eMAG) / 20.04, 21.04 (Rpi4) clang/llvm 12.0.0 , 12.0.1, 13.0.0
It is still possible to point PATH to the prebuilt toolchain version (Android llvm/clang 12.0.5) as indicated in the documentation.
If you have a live tree, please clean the out directory or run make clobber, once you update master. Builds run as before after the switch.
Limitations:
-the build breaks if using a native toolchain installed on the host (apt-get install clang..)
-the build breaks with Ubuntu 21.10/AArch64 (under investigation).
Regards,
Olivier.
TF-A List. This issue has also been discussed on Hafnium list before being posted here. Cross posting so we can have a single thread to track going forward.
See https://lists.trustedfirmware.org/pipermail/hafnium/2021-December/000209.ht…
with Olivier's last reply copied below. But see the archive above for full history of the thread.
> Hi Wang,
>
> With this level of details; this is difficult to say. You can extend to the TF-A ML if you wish. I'm hinting the SPMD because you are mentioning spmd_smc_forward and cm_el1/2_sysregs_context_restore which are within the SPMD/EL3 space. I wouldn't expect such assert to happen in any regular use case of the reference implementation (because this is a hard EL3 failure). But yes, the problem can be elsewhere in Hafnium or Cactus, but I'd say less likely to alter the EL3 state. Unless Hafnium has a bug leading to corrupting a secure memory region which doesn't belong to it.
> Beyond this, notice the assert is taken in cm_el1_sysregs_context_restore. It is called by cm_prepare_el3_exit which means it can be related to power management e.g. on a psci resume event. This can be a hint as you say this is occurring 'randomly'.
>
> Regards,
> Olivier.
Joanna
On 14/12/2021, 19:39, "TF-A on behalf of Chenxu Wang via TF-A" <tf-a-bounces(a)lists.trustedfirmware.org on behalf of tf-a(a)lists.trustedfirmware.org> wrote:
Hi all,
I am running FVP with 2CPUs, Cactus SP (SEL1), Hafnium (SEL2) and KVM VHE.
Sometimes I send the "FFA_MSG_SEND_DIRECT_REQ" smc call from KVM (I fill
0x8400006f in x0, then VMID and SP ID in x1, let x2 as 0). It says
assert failed, like this:
ASSERT: lib/el3_runtime/aarch64/context_mgmt.c:651
BACKTRACE: START: assert
0: EL3: 0x4005cac
1: EL3: 0x400323c
2: EL3: 0x400620c
3: EL3: 0x400e180
4: EL3: 0x4005a94
BACKTRACE: END: assert
After I check the bl31.dump, I notice that:
when services/std_svc/spmd/spmd_main.c sends the FFA
call (from NS to S) via "spmd_smc_forward(smc_fid, secure_origin,x1,
x2, x3, x4, handle)", it will go to
cm_el1_sysregs_context_restore(secure_state_out) and
cm_el2_sysregs_context_restore(secure_state_out), then it will assert
the cm_get_context(). it gets the NULL context, so assert failed.
Before the problem appeared, I have modified many codes on a dirty
TF-A v2.4 (commit hash is 0aa70f4c4c023ca58dea2d093d3c08c69b652113),
Hafnium and TF-A-TESTS. I also mail with Hafnium MailList, they
consider it can be a problem in EL3.
Such assert is NOT ALWAYS failed. I mean, maybe when I run FVP and
send "smc" now, it is failed. But when I shut down, run FVP, and send
the same instruction with the same parameter again, it is OK.
I want to know, what is the possible reasons for suddenly losing the
secure context. Can you give me some advice on debugging? e.g., where
should I check? Need I provide more info?
Sincerely,
Wang
--
TF-A mailing list
TF-A(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-a
Hi all,
I am running Hafnium on FVP, with Cactus SP in SEL1 and KVM VHE enabled.
Sometimes I send the "FFA_MSG_SEND_DIRECT_REQ" smc call in KVM (I fill
0x8400006f in x0, then VMID and SP ID in x1, let x2 as 0). It says
assert failed, like this:
ASSERT: lib/el3_runtime/aarch64/context_mgmt.c:651
BACKTRACE: START: assert
0: EL3: 0x4005cac
1: EL3: 0x400323c
2: EL3: 0x400620c
3: EL3: 0x400e180
4: EL3: 0x4005a94
BACKTRACE: END: assert
I notice that when services/std_svc/spmd/spmd_main.c sends the FFA
call (from NS to S) via "spmd_smc_forward(smc_fid, secure_origin,x1,
x2, x3, x4, handle)", it will go to
cm_el1_sysregs_context_restore(secure_state_out) and
cm_el2_sysregs_context_restore(secure_state_out), then it will assert
the cm_get_context(). it gets the NULL context, so assert failed.
Such assert is NOT ALWAYS failed, but I still want to solve this problem.
Since I have modified many lines of code in Hafnium and Cactus SP, I
cannot show them here. Can you give me some advice on debugging? e.g.,
where should I check?
Hi Xiangyi Xu
For some reason your email was discarded by mailman, did you miss registering to the list?
See few comments inline [OD]
Regards,
Olivier.
________________________________________
From: xiangyi xu <xuxiangyi666(a)gmail.com>
Sent: 27 November 2021 14:13
To: hafnium(a)lists.trustedfirmware.org
Subject: Virtualising OP-TEE with Hafnium at S-EL2
HI ALL:
I am trying to setup Hafnium environment which loads OP-TEE in Security World and Linux in Normal World. I follow this instruction: https://lists.trustedfirmware.org/pipermail/hafnium/2021-January/000130.html. I can load the OP-TEE test successfully. But the BL33 payload is TF-A-tests example (BL33=../tf-a-tests/build/fvp/debug/tftf.bin). Could you share with us the tutorial to boot linux including the FF-A driver in the NS world while loading OP-TEE with Hafnium S-EL2? The presentation PPT is here: https://static.linaro.org/connect/lvc21/presentations/lvc21-305.pdf.
[OD] The mentioned presentation is in context of the Total Compute platform (as opposed to FVP).
It works great for what you want to do, and you can reproduce by following the user manual:
https://gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-docs…
There is also work in progress at Linaro to provide a similar environment using the FVP.
@Jens, are there public instructions on how to build the end to end scenario targeting the FVP incl. Linux, FF-A driver, TF-A, Hafnium, OP-TEE ?
If not, I reckon this is in your mid term roadmap?
Secondly, I can boot Linux with QEMU and Hafnium. But it seems Hafnium only supports NS world partition while using QEMU. Is any instruction to boot OP_TEE in Security world with Hafnium S-EL2. Since, QEMU 6.0 already supports S-EL2. However, I could not compile ATF successfully with SPD=spmd CTX_INCLUDE_EL2_REGS=1 PLAT=qemu ...
[OD] qemu is not 'officially' supported for booting Hafnium at S-EL2. Although I have few changes in progress which may help. This is not a priority from Arm perspective, but let me come back on this.
By following https://teaclave.apache.org/trustzone-sdk-docs/getting-started-with-optee-f… , I can boot Linux in NS and OP-TEE in Trustzone with QEMU, but it doesn't work when I add Hafnium. Thanks!
Xiangyi Xu
