Hi Raghu,
Comments inline [OD]
Regards, Olivier.
________________________________________ From: raghu.ncstate@icloud.com raghu.ncstate@icloud.com Sent: 15 December 2020 18:41 To: Olivier Deprez; hafnium@lists.trustedfirmware.org; Achin Gupta Cc: Mayur Gudmeti Subject: RE: [Hafnium] VHE support
Hi Olivier,
Sorry if my questions below sound obvious, I may miss bits of the VHE
architecture.
[RK] Thanks. Please feel free to push me on these questions. This was done more or less in isolation so I'm happy to be corrected and my understanding checked. Please feel free to suggest alternative paths to take as well.
[OD] Sure.
I guess it requires an additional change in project/reference to
eventually enable the feature for a platform (enable_vhe=1 in the gn build flow)?
[RK] Correct. I had this change locally and did not push it since the default would be to have enable_vhe=0
The changes are effectively toggling HCR_EL2.E2H=1. Currently when
Hafnium ERETs resuming a SP, this happens with the secure EL1&0 translation regime. Though what's the next step forward? Does this require a host OS in secure world? (like linux does when booting under a Hypervisor with VHE enabled?) Would this be a VHE-enabled TOS?
[RK] Hanfium effectively becomes the host OS when we enable E2H, so we don't need anything special. Sorry if this is obvious or you already know - When linux boots with VHE enabled, it runs code written to work on EL1 in EL2 as the host OS and and EL1 register accesses redirect to the equivalent EL2 registers. Hafnium is effectively doing this, except hafnium's code is already written to access the EL2 system registers so I think of hafnium as the host OS. Let me know if that does not make sense, we can discuss further. The next steps answered below.
[OD] Yes, I got this. So in other words, the Hypervisor and the Host OS are conflated in this configuration.
Is the later goal to enable the secure EL2&0 translation regime? 4/ Is there anything to do with HCR_EL2.TGE?
[RK] EL2&0 translation regime is enabled as soon as we set E2H is set, so the current patches already enable it. Note that I'm only using TTBR0_EL2 and not using TTBR1_EL2 since there is no need for it as of today. I don't expect there to be use for it in the near future either given that we want to use identity mapping everywhere.
[OD] Right, so my understanding is that with this patch set Hafnium runs the EL2&0 translation regime. Although EL1/EL0 accesses are still made using the EL1&0 translation regime because HCR_EL2.TGE=0.
As for future steps, this is what I'm thinking - note that this needs more thought and exploration and are not fool proof thoughts: 1) Model S-EL0/EL0 partitions as "lightweight" VM's. By lightweight, what I mean is that we represent EL0 partitions using the existing VM and VCPU structures, except these VM's will be forced to only have a maximum of one VCPU,
[OD] Just curious, why restricting to a single vCPU? Is this to map with the existing StMM model? Anyways, there is always the possibility to set the VM's vCPU count to 1 in the SPMC manifest.
and the context switch for these partitions would be "light weight", ie only GPR's and tbbr0_el2. I have to explore some of the other settings/system registers in EL2 that may have to be set up appropriately and is open. The advantage of doing so is that we can reuse all the loading infrastructure for VM's on EL0 partitions too.
[OD] Right, notice there is a provision in partition manifests to state that a partitions is an S-EL0 partition (exception-level field set to S_EL0 in https://trustedfirmware-a.readthedocs.io/en/latest/components/psa-ffa-manife...) This property can be used to identify this configuration.
2) As for TGE, HCR_EL2.TGE will be set only for EL0 partitions so that these partitions's would become the equivalent of "host" applications. This bit will not be set for S-EL1 partitions/VM's. Effectively, when E2H is set, this bit differentiates between a VM and a host application.
[OD] What I still miss is how the VM isolation is done. If a S-EL0 partition is a host application, does it mean page tables are shared between the Hypervisor/Host OS and the S-EL0 partition? This looks to provide more privileges to S-EL0 partitions than regular S-EL1 SPs? Or in other words, memory accesses for the applicative part only go through one translation stage (the Stage-1 EL2&0) and not through the 2-stage MMU? Or maybe the model is that there is a unique more privileged S-EL0 SP?
IIUC if we take linux, the host OS executes with E2H/TGE=11 (both set early at boot time). The KVM infrastructure provides the interface to create/run a VM which will then execute with TGE=0. So to make the parallel with secure Hafnium should we have TGE=1 from an early start, and toggle it to zero when switching to a S-EL1 SP?
3) There needs to be more thought around how hafnium will handle EL0 partitions vs VM's w.r.t to interrupts since there is no injection of virtual interrupts in this case. Exception handling and TLB maintenance are other area's I need to explore.
[OD] With TGE=1 exceptions and interrupts would still trap at (S)EL2 There is still the possibility a secure interrupt is injected to an S-EL1 SP, or that a managed exit is required. Btw if I recall the model for StMM is run-to-completion without interrupt?
4) Anything else that comes up?? I plan to start prototyping this and see where it takes me but at this point it seems achievable without having to break hafnium entirely.
Hi Achin,
Question for you. I interpreted the FF-A 1.0 spec as "requiring" VHE for S-EL0 partitions, based on options 1 and 2 in section 2.2.1, since option 2 does not mention S-EL0 partitions. Can you confirm this is the case? As I was thinking through this, it seems like we might be able to pull off EL0 partitions without using VHE, i.e by using HCR_EL2.E2H = 0 and HCR_EL2.TGE=1. HCR_EL2.TGE is present even on ARMv8.0 architecture and removes dependence on VHE, and effectively removes the necessity for this patch series. However, I'd like to understand your thoughts on why VHE is suggested in the spec and if hafnium should use VHE for S-EL0 partitions or if hafnium can use the approach suggested above.
Thanks Raghu
-----Original Message----- From: Olivier Deprez Olivier.Deprez@arm.com Sent: Tuesday, December 15, 2020 12:18 AM To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com; Olivier Deprez Olivier.Deprez@arm.com Subject: Re: [Hafnium] VHE support
Hi Raghu,
one more
5/ maybe answer to 2/3/4 is that it requires an EL1-shim embedded into Hafnium which itself ERETs to a S-EL0 partition?
BTW notice my questions are obviously oriented towards the secure implementation.
Regards, Olivier.
________________________________________ From: Hafnium hafnium-bounces@lists.trustedfirmware.org on behalf of Olivier Deprez via Hafnium hafnium@lists.trustedfirmware.org Sent: 15 December 2020 09:10 To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com Subject: Re: [Hafnium] VHE support
Hi Raghu,
Thanks for sharing this work.
Few thoughts...
1/ I guess it requires an additional change in project/reference to eventually enable the feature for a platform (enable_vhe=1 in the gn build flow)?
Sorry if my questions below sound obvious, I may miss bits of the VHE architecture.
2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when Hafnium ERETs resuming a SP, this happens with the secure EL1&0 translation regime. Though what's the next step forward? Does this require a host OS in secure world? (like linux does when booting under a Hypervisor with VHE enabled?) Would this be a VHE-enabled TOS?
3/ Is the later goal to enable the secure EL2&0 translation regime?
4/ Is there anything to do with HCR_EL2.TGE?
Regards, Olivier.
________________________________________ From: Hafnium hafnium-bounces@lists.trustedfirmware.org on behalf of Raghu Krishnamurthy via Hafnium hafnium@lists.trustedfirmware.org Sent: 15 December 2020 04:57 To: hafnium@lists.trustedfirmware.org Subject: [Hafnium] VHE support
Hi All,
I have a series of patches pushed to Gerrit at https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599 with topic "vhe_enable". The goal of this patch series is to enable VM's in both secure and normal world to run with VHE enabled(hcr_el2.e2h=1), without breaking any existing functionality. This is expected to be the first step in the long term goal of enabling S-EL0 partitions(and optionally EL0 partitions), that require VHE support, per the FF-A 1.0 Spec. I'd appreciate feedback on the patches and approach taken to nominally enabling VHE. Note that the FF-A 1.0 spec(AFAIK) does not expect VHE support in the normal world but this patch series enables it anyway due to the wealth of available tests in the hafnium test suite to help with providing confidence in the implementation.
The patch series has been tested as follows:
Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE, since the prebuilt QEMU does not support VHE.
Hafnium tests using QEMU(5.2-RC4, built from source) - With and without VHE, this version of QEMU supports VHE.
Hafnium tests using FVP 11.12.28 - With and Without VHE.
TFTF tests for secure hafnium using FVP 11.12.28 - With and without VHE.
Thanks
Raghu
-- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium -- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium