+ ML
________________________________________ From: Olivier Deprez Olivier.Deprez@arm.com Sent: 03 January 2022 15:07 To: Bo Yan; raghu.ncstate@icloud.com Cc: Varun Wadekar Subject: Re: [Hafnium] .git submodules increase hafnium code size
Hi Bo, Raghu,
I knew this change would be a bit controversial :) We've had the following main queries:
1/ submodule(s) size is too large (see Varun's very first message in this thread). Starting with prebuilts, it turns out that llvm+gnu toolchains are top contributors. The mentioned change (on top of earlier clang migration changes) introduced the ability to remove a hard dependency to those toolchains. 2/ Ability to build on arm64 host. The change removes the compiler/linker/tools hard coded paths to the x86 prebuilt toolchain from the build system. 3/ toolchain is hardcoded to a fixed version and buried into the project along with sources. This fits developer habits but it is not suitable to a production environment.
Addressing few of the concerns below:
The default built at the top of tree seems broken. Typing make does not work any more. I tried to export the clang path and I see issues with standard headers...docs/GettingStarted is incomplete?
This is the glitch I expected to happen when pulling latest changes. Hence the reason for the earlier heads up on the ML. Hopefully this recovers after cleaning the tree once, or restarting from a freshly cloned tree.
I'm not sure I like the idea of removing the prebuilts. We need one tool chain that we know is supported and tested with to create reproducible builds.
ATM the CI still points to the hafnium prebuilt toolchain. Hence builds are still 'reproducible'. It could be proposed that the CI environment itself provides the toolchain.
We can remove the dependency on the prebuilt in that it should be possible for someone to NOT require downloading the upstream tool chain if they want to use an alternate chain,
I get the idea, but we have two orthogonal asks (use an external toolchain and remove from prebuilts, or keep in prebuilt but clone conditionally). The .gitmodules does not provide this flexibility AFAIK. I was thinking of using some partial clone command like below omitting prebuilts: git submodule update --init driver/linux project/reference third_party/dtc third_party/googletest third_party/linux
(but this doesn't look great from an UX perspective).
for reference platforms like FVP, the toolchain should be a part of the build
Not sure this is a hard dependency. Comparing with TF-A (or other firmware projects), the toolchain is certainly not held within the tree. The developer or the CI environment provides the toolchain to build the project.
The current method was ideal but it seems like some latest changes have broken the dev flow...
Ideal for a developer sticking to the provided fixed-version android-based prebuilt toolchain, and building on an x86 host. Not ideal if one needs to build on arm64, or in a production build when hafnium is considered one component among many others within a (yocto-like) distribution.
I agree on the point that keeping prebuilts brings benefits, but the build system should be flexible enough such that an out-of-tree toolchain can be easily selected.
This is what this first set of changes provide.
Regards, Olivier.
________________________________________ From: Bo Yan byan@nvidia.com Sent: 21 December 2021 00:45 To: raghu.ncstate@icloud.com; Varun Wadekar; Olivier Deprez Cc: 'Raghu Krishnamurthy via Hafnium' Subject: RE: [Hafnium] .git submodules increase hafnium code size
I agree on the point that keeping prebuilts brings benefits, but the build system should be flexible enough such that an out-of-tree toolchain can be easily selected.
-----Original Message----- From: raghu.ncstate@icloud.com raghu.ncstate@icloud.com Sent: Saturday, December 18, 2021 1:22 PM To: raghu.ncstate@icloud.com; Varun Wadekar vwadekar@nvidia.com; 'Olivier Deprez' Olivier.Deprez@arm.com Cc: Bo Yan byan@nvidia.com; 'Raghu Krishnamurthy via Hafnium' hafnium@lists.trustedfirmware.org Subject: RE: [Hafnium] .git submodules increase hafnium code size
Never mind. Sorry for the false alarm. Had to use make clobber( found out once I caught up on the email list...). Other comments about continuing to have a prebuilts directory like we have today still apply.
-----Original Message----- From: Hafnium hafnium-bounces@lists.trustedfirmware.org On Behalf Of Raghu Krishnamurthy via Hafnium Sent: Saturday, December 18, 2021 1:08 PM To: 'Varun Wadekar' vwadekar@nvidia.com; 'Olivier Deprez' Olivier.Deprez@arm.com Cc: 'Bo Yan' byan@nvidia.com; 'Raghu Krishnamurthy via Hafnium' hafnium@lists.trustedfirmware.org Subject: Re: [Hafnium] .git submodules increase hafnium code size
Add hafnium list.
Fix for broken build at - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Frevie w.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium%2F%2B%2F13150&am p;data=04%7C01%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26 c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63775459320 6358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9C%2F0A6 1IlusyOnozlGCclZN13YPjaRm1TenBDpemK8M%3D&reserved=0
-----Original Message----- From: raghu.ncstate@icloud.com raghu.ncstate@icloud.com Sent: Saturday, December 18, 2021 12:51 PM To: 'Varun Wadekar' vwadekar@nvidia.com; 'Olivier Deprez' Olivier.Deprez@arm.com Cc: 'Bo Yan' byan@nvidia.com Subject: RE: [Hafnium] .git submodules increase hafnium code size
The default built at the top of tree seems broken. Typing make does not work any more. I tried to export the clang path and I see issues with standard headers...docs/GettingStarted is incomplete? I'm not sure I like the idea of removing the prebuilts. We need one tool chain that we know is supported and tested with to create reproducible builds. We can remove the dependency on the prebuilt in that it should be possible for someone to NOT require downloading the upstream tool chain if they want to use an alternate chain, but for reference platforms like FVP, the toolchain should be a part of the build. The current method was ideal but it seems like some latest changes have broken the dev flow...
-----Original Message----- From: Hafnium hafnium-bounces@lists.trustedfirmware.org On Behalf Of Varun Wadekar via Hafnium Sent: Friday, December 17, 2021 7:58 AM To: Olivier Deprez Olivier.Deprez@arm.com Cc: Bo Yan byan@nvidia.com; hafnium@lists.trustedfirmware.org Subject: Re: [Hafnium] .git submodules increase hafnium code size
Thanks Olivier. Makes sense to me. We are deploying these changes to our build system. Will let you know.
-----Original Message----- From: Olivier Deprez Olivier.Deprez@arm.com Sent: Friday, 17 December 2021 3:32 PM To: Varun Wadekar vwadekar@nvidia.com Cc: Bo Yan byan@nvidia.com; hafnium@lists.trustedfirmware.org Subject: Re: .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Varun,
I don't believe the prebuilt submodule would be fully removed. There are libraries and binaries required by the build in any case in the linux-aarch64 directory. I rather thought the toolchains within it might be removed. They are the main contributor in terms of disk space savings which is the problem if I understood properly.
As an intermediate step, before removing the toolchain, I would like to confirm that you are able to switch to another toolchain and there is no longer a dependency to the toolchain stored in prebuilt.
Regards, Olivier.
From: Varun Wadekar vwadekar@nvidia.com Sent: 17 December 2021 15:18 To: Olivier Deprez Cc: Bo Yan; hafnium@lists.trustedfirmware.org Subject: RE: .git submodules increase hafnium code size
Thanks Olivier. We hit the toolchain changes yesterday and still navigating a course to enable it internally.
When do you plan to remove the prebuilt submodule completely?
-----Original Message----- From: Olivier Deprez Olivier.Deprez@arm.com Sent: Friday, 17 December 2021 12:00 PM To: Varun Wadekar vwadekar@nvidia.com Cc: Bo Yan byan@nvidia.com; hafnium@lists.trustedfirmware.org Subject: Re: .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Varun,
Just a quick sync on what's been going since this last email.
A number of changes have been made to: -remove the gcc dependency, hafnium now only requires clang to build. -ability to provide a toolchain different from the one stored in prebuilt submodule (merged yesterday) -migrated and tested with recent clang versions (was clang 9) -ability to build on arm64 host (our need)
The prebuilt submodule still exists but the dependency to hardcoded toolchains has weakened.
I intend to tell more about those recent changes and what remains during a tech forum early next year, among: -remove toolchains from prebuilt? -split the hypervisor and spmc test configs into separate project/* directories -lessen the dependency to other 3rd party projects
Regards, Olivier.
From: Hafnium hafnium-bounces@lists.trustedfirmware.org on behalf of Olivier Deprez via Hafnium hafnium@lists.trustedfirmware.org Sent: 08 June 2021 16:00 To: Varun Wadekar Cc: Bo Yan; hafnium@lists.trustedfirmware.org Subject: Re: [Hafnium] .git submodules increase hafnium code size
Hi Varun,
As indicated earlier, we (Arm side) don't expect to progress on this front before Q3.
I'm gathering requirements and expect to discuss through a short presentation later in a tech forum or the ML.
Regards, Olivier.
From: Varun Wadekar vwadekar@nvidia.com Sent: 08 June 2021 15:21 To: Arunachalam Ganapathy; Olivier Deprez Cc: Bo Yan; hafnium@lists.trustedfirmware.org Subject: RE: .git submodules increase hafnium code size
Thanks Arun. We have taken the same approach. The changes I posted earlier expect all dependencies to be out of tree and provide mechanisms to pass the locations to the make system during compilation.
This is a real problem for us, and we would like a solution that works for the community too. @Olivier how should we move forward?
-----Original Message----- From: Arunachalam Ganapathy Arunachalam.Ganapathy@arm.com Sent: Tuesday, June 8, 2021 2:08 PM To: Varun Wadekar vwadekar@nvidia.com; Olivier Deprez Olivier.Deprez@arm.com Cc: Bo Yan byan@nvidia.com; hafnium@lists.trustedfirmware.org Subject: Re: .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Varun,
1- First in context of Total Compute delivery from Arm OSS platforms: a. ability to build only the SPMC on TC0 platform (not all reference
targets such as qemu, rpi4, fvp)
b. use a Yocto provided toolchain. @Arun, your view on how those two items were solved is beneficial tofurther elaborate our plans.
For Total Compute we wanted to skip cloning submodules (like driver/linux, linux, dtc) as only secure_hafnium.bin was required. Basically build only reference spm. Like: make PROJECT=reference_spm PLAT=TC this builds only secure hafnium for one platform.
There were some efforts put on 1.a and was able to build reference_spm for one platform. But Hafnium build inside yocto forced to clone all submodules (due to dependencies on prebuilt toolchains), so the changes were dropped and also the changes weren't clean enough to be upstreamed. Regarding 1.b we didn't try hafnium build using yocto toolchain.
Thanks, Arun
-----Original Message----- From: Varun Wadekar vwadekar@nvidia.com Sent: Monday, June 7, 2021 14:43 To: Varun Wadekar vwadekar@nvidia.com; Olivier Deprez Olivier.Deprez@arm.com; Arunachalam Ganapathy Arunachalam.Ganapathy@arm.com Cc: Bo Yan byan@nvidia.com; hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org Subject: RE: .git submodules increase hafnium code size
Hi,
@Arun, your view on how those two items were solved is beneficial to
further elaborate our plans.
@Arunachalam Ganapathy your comments on this topic would be very helpful.
Thanks.
-----Original Message----- From: Hafnium hafnium-bounces@lists.trustedfirmware.org On Behalf Of Varun Wadekar via Hafnium Sent: Monday, May 31, 2021 1:49 PM To: Olivier Deprez Olivier.Deprez@arm.com; hafnium@lists.trustedfirmware.org; Arunachalam Ganapathy Arunachalam.Ganapathy@arm.com Cc: Bo Yan byan@nvidia.com Subject: Re: [Hafnium] .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Olivier,
Thanks for answering my queries.
We are looking to deploy the following use case at NVIDIA.
<snip>
-ability to build only the SPMC (not all reference targets such as qemu, rpi4, fvp) -A distribution only requiring the Hypervisor/SPMC output binary ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or x86 host, and arbitrary clang version).
<snip>
As you noticed, the Hafnium Hypervisor/SPMC and test environment
builds are closely coupled by the use of ninja/gn flow and scripts. We intend to approach those problems in the course of Q3 in Arm OSS roadmap.
[VW] Are there any local changes to decouple hafnium from its dependencies? We can evaluate Arm;s approach against what we use internally. Our changes moved the dependencies out of the tree and passed file locations to the build system with the help of command line arguments.
-Varun
-----Original Message----- From: Olivier Deprez Olivier.Deprez@arm.com Sent: Monday, May 31, 2021 11:03 AM To: hafnium@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com; Arunachalam Ganapathy Arunachalam.Ganapathy@arm.com Cc: Bo Yan byan@nvidia.com Subject: Re: .git submodules increase hafnium code size
External email: Use caution opening links or attachments
Hi Varun,
We had similar requests raised internally. 1- First in context of Total Compute delivery from Arm OSS platforms: a. ability to build only the SPMC on TC0 platform (not all reference targets such as qemu, rpi4, fvp) b. use a Yocto provided toolchain. @Arun, your view on how those two items were solved is beneficial to further elaborate our plans. 2- A similar request as 1.b to build Hafnium as part of a distribution on arm64 host: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeve loper.trustedfirmware.org%2FT898&data=04%7C01%7Cbyan%40nvidia. com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39 efd9ccc17a%7C0%7C0%7C637754593206358976%7CUnknown%7CTWFpbGZs b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn 0%3D%7C3000&sdata=AtMec%2BNFfhA14xeMZT992icEj6SLAhwLnB2co5 VXFNM%3D&reserved=0
In my view there are two consumers: -A distribution only requiring the Hypervisor/SPMC output binary ("out/reference/.../hafnium.bin") using any toolchain (be it arm64 or x86 host, and arbitrary clang version). -The Hf CI framework/automation needs the above, plus the test framework and tests (dependency on googletest, linux submodules etc). It's important to keep this item alive while trying to solve above item.
As you noticed, the Hafnium Hypervisor/SPMC and test environment builds are closely coupled by the use of ninja/gn flow and scripts. They are using a fixed toolchain version through prebuilts to ensure builds are "reproducible", in particular with regards to the Hafnium CI.
We intend to approach those problems in the course of Q3 in Arm OSS roadmap. As an early exploration we already have: -clang 12 compiler upgrade. This is necessary if wiling to use any arbitrary clang version: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Frevie w.trustedfirmware.org%2Fq%2Ftopic%3A%2522od%25252Fhf- clang12%2522&data=04%7C01%7Cbyan%40nvidia.com%7C030b52fc550 94b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C 0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000& ;sdata=t7KJjvv2o8EHKBpPdUEIrJ%2F7Vy7KFEODdnQMrL4LpaU%3D&res erved=0+(status:open%20OR%20status:merged) -Ability to build on arm64 host (done, internally). -Identify the flow/script changes such that external dependencies can be used (on-going, internally). I thought of localizing common dependencies to python/shell scripts by the use of definition files included in the mentioned scripts. This is only an early investigation, I will check how this intersects the changes you provided.
Regards, Olivier.
From: Hafnium hafnium-bounces@lists.trustedfirmware.org on behalf of Varun Wadekar via Hafnium hafnium@lists.trustedfirmware.org Sent: 28 May 2021 16:47 To: hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org Cc: Bo Yan byan@nvidia.com Subject: [Hafnium] .git submodules increase hafnium code size
Hi,
We at NVIDIA are evaluating Hafnium. During the initial investigation, we found out that the repository size (in terms of MB) is huge. This is mostly because of the "git submodules" used by the project. This is a great way to deliver Hafnium with its dependencies in one go.
But we think that the size can be trimmed by moving the toolchain, linux folder, googletest and dtc compiler out, leaving just the Hafnium code in the project. This way, companies like us can pick and choose instead of having to use everything. In a bid to ease the pain internally and only use the Hafnium code base we have crafted the following changes:
- hafnium: support external projects (I10a07de3) * Gerrit Code Review
(trustedfirmware.org)https://nam11.safelinks.protection.outlook.com/?url =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium %2F%2B%2F10142&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0 %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000& amp;sdata=7PWBcTXxP6HtqN%2BTGz3cEnnfabSzlyjo0RNZOjvOL4A%3D& ;reserved=0 2. hafnium: build with dtc and googletest out of tree (I057c9ad6) * Gerrit Code Review (trustedfirmware.org)https://nam11.safelinks.protection.outlook.com/?url =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium %2F%2B%2F10144&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0 %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000& amp;sdata=Ql8XW8FAEQ%2FereVjB3glccWJLR7sQ3yg0v9%2FSptCnyk%3D&a mp;reserved=0 3. build: support external toolchain (Iafd029c1) * Gerrit Code Review (trustedfirmware.org)https://nam11.safelinks.protection.outlook.com/?url =https%3A%2F%2Freview.trustedfirmware.org%2Fc%2Fhafnium%2Fhafnium %2F%2B%2F10145&data=04%7C01%7Cbyan%40nvidia.com%7C030b52f c55094b6fa3b008d9c26c6bdf%7C43083d15727340c1b7db39efd9ccc17a%7C0 %7C0%7C637754593206358976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000& amp;sdata=X%2FNE1WmrcQvqRzEBq8UXdA02y9rl%2BS%2Fs1btuRAAlacM%3 D&reserved=0
This series does not have the patch to use an out of tree linux codebase. I assume these patches wont be acceptable in their current state, so would like to know how the community plans to handle this situation.
The code size is a real concern for us, as we already have copies of the dependencies in our codebase, so have no use for these duplicates.
Thanks.
Hafnium mailing list Hafnium@lists.trustedfirmware.org https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0 -- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0 -- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0 -- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0
-- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.t rustedfirmware.org%2Fmailman%2Flistinfo%2Fhafnium&data=04%7C0 1%7Cbyan%40nvidia.com%7C030b52fc55094b6fa3b008d9c26c6bdf%7C4308 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637754593206358976%7CUn known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I k1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tBM%2FCZq%2FjMDpp60Y ZN5sounTBfkkRjylu%2Fes3IpZBaE%3D&reserved=0