Hi All,
I was going through src/mm.c in the hafnium repo and noticed a couple of
things that I think are problem:
1. mm_vm_defrag and mm_defrag, which eventually call mm_ptable_defrag
appears to be changing page entries into block entries. Per D5.10.1 of ARM
ARM, this should use break-before-make, but I don't see it being done in
code. Am I missing where this is done in code, or is there some reason
break-before-make is not required during defrag(that is not obvious to me
from code or comments)?
2. In mm_replace_entry, line 278
(https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/mm.c#n278) is
not followed by a DSB + ISB as required by the break-before-make. Without
the DSB, a load/store that uses a VA translated by the PTE being replaced
could be reordered before line 278 and fault. Once again, this looks like a
bug to me but is there some reason that is not obvious due to which it does
not require barriers?
Thanks
Raghu
Hi Raghu,
one more
5/ maybe answer to 2/3/4 is that it requires an EL1-shim embedded into Hafnium which itself ERETs to a S-EL0 partition?
BTW notice my questions are obviously oriented towards the secure implementation.
Regards,
Olivier.
________________________________________
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 15 December 2020 09:10
To: hafnium(a)lists.trustedfirmware.org; raghu.ncstate(a)icloud.com
Subject: Re: [Hafnium] VHE support
Hi Raghu,
Thanks for sharing this work.
Few thoughts...
1/ I guess it requires an additional change in project/reference to eventually enable the feature for a platform (enable_vhe=1 in the gn build flow)?
Sorry if my questions below sound obvious, I may miss bits of the VHE architecture.
2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when Hafnium ERETs resuming a SP, this happens with the secure EL1&0 translation regime.
Though what's the next step forward?
Does this require a host OS in secure world? (like linux does when booting under a Hypervisor with VHE enabled?) Would this be a VHE-enabled TOS?
3/ Is the later goal to enable the secure EL2&0 translation regime?
4/ Is there anything to do with HCR_EL2.TGE?
Regards,
Olivier.
________________________________________
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Raghu Krishnamurthy via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 15 December 2020 04:57
To: hafnium(a)lists.trustedfirmware.org
Subject: [Hafnium] VHE support
Hi All,
I have a series of patches pushed to Gerrit at
https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599 with topic
"vhe_enable". The goal of this patch series is to enable VM's in both secure
and normal world to run with VHE enabled(hcr_el2.e2h=1), without breaking
any existing functionality. This is expected to be the first step in the
long term goal of enabling S-EL0 partitions(and optionally EL0 partitions),
that require VHE support, per the FF-A 1.0 Spec. I'd appreciate feedback on
the patches and approach taken to nominally enabling VHE. Note that the FF-A
1.0 spec(AFAIK) does not expect VHE support in the normal world but this
patch series enables it anyway due to the wealth of available tests in the
hafnium test suite to help with providing confidence in the implementation.
The patch series has been tested as follows:
Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE, since
the prebuilt QEMU does not support VHE.
Hafnium tests using QEMU(5.2-RC4, built from source) - With and without VHE,
this version of QEMU supports VHE.
Hafnium tests using FVP 11.12.28 - With and Without VHE.
TFTF tests for secure hafnium using FVP 11.12.28 - With and without VHE.
Thanks
Raghu
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
Hi Achin,
> HCR_EL2.{E2H,TGE} = {1,1} reduces the number of translation stages managed by
> the SPMC for S-EL0 SPs. It allows them to be managed like apps instead of VMs
> which is the case pre-v8.4. I think this is the better configuration for a
> system that only deploys S-EL0 SPs.
If the system only deploys S-EL0 SPs in this configuration, what's the benefit of using Hafnium in place of a S-EL1 TEE?
(appart maybe from product / SW scalability perspective?)
Regards,
Olivier.
________________________________________
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Achin Gupta via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 19 December 2020 15:24
To: raghu.ncstate(a)icloud.com
Cc: hafnium(a)lists.trustedfirmware.org; 'Mayur Gudmeti'; 'Joakim Bech'
Subject: Re: [Hafnium] VHE support
Hi Raghu,
On Fri, Dec 18, 2020 at 02:39:44PM -0800, raghu.ncstate(a)icloud.com wrote:
> Thanks Achin.
>
> >> Now if a S-EL0 SP is run in this configuration, irrespective of how TGE is configured, the SPMC would have to manage both the Stage 1 and 2 translations on behalf of the S-EL0 SP in the Secure EL1&0 translation regime,
> [RK] Not sure I understand the statement "irrespective of how TGE is configured". If HCR_EL2.E2H,TGE=1,1, then SPMC would manage Secure EL2&0 translation regime for the S-EL0 SP and crucially, no stage 2 translations. Your statement applies only when HCR_EL2.{E2H,TGE} = {1,1}. The statement right after this contradicts "irrespective of how TGE is configured".
Apologies. I meant E2H and not TGE i.e. if HCR_EL2.TGE=0, then the SPMC has to
manage both the Stage 1 and 2 translations on behalf of a S-EL0 SP in the
Secure EL1&0 translation regime,
>
> >> Management of a S-EL0 SP in the earlier configuration is expected to be easier.
> [RK] Sorry, which configuration? Earlier is vague.
HCR_EL2.{E2H,TGE} = {1,1}
>
> >> Does this seem reasonable?
> Yes, this is reasonable. However, what configuration of HCR_EL2.{E2H, TGE} should hafnium use to manage S-EL0 SP's? Should it use {0, 1} (not in spec) or {1, 1}( Section 2.2.1, Bullet 1)?
HCR_EL2.{E2H,TGE} = {1,1} reduces the number of translation stages managed by
the SPMC for S-EL0 SPs. It allows them to be managed like apps instead of VMs
which is the case pre-v8.4. I think this is the better configuration for a
system that only deploys S-EL0 SPs.
The spec does not specify the values of E2H or TGE. So {0, 1} is not disallowed
by the spec either.
Looks like that the choices we have for Hafnium are:
1. {1, 1} for S-EL0 SPs and {1, 0} for S-EL1 SPs
2. {0, 1} for S-EL0 SPs and {0, 0} for S-EL1 SPs
We would need to understand the impact of these choices on the Hafnium
implementation before making a call.
The v1.1 spec can recommend the sensible choices to avoid confusion.
cheers,
Achin
>
> Thanks
> Raghu
>
> -----Original Message-----
> From: Achin Gupta <achin.gupta(a)arm.com>
> Sent: Friday, December 18, 2020 1:57 PM
> To: François Ozog <francois.ozog(a)linaro.org>; raghu.ncstate(a)icloud.com
> Cc: raghu.ncstate(a)icloud.com; hafnium(a)lists.trustedfirmware.org; Mayur Gudmeti <mgudmeti(a)nvidia.com>; Ruchika Gupta <ruchika.gupta(a)linaro.org>; Joakim Bech <joakim.bech(a)linaro.org>
> Subject: Re: [Hafnium] VHE support
>
> Hi Francois and Raghu,
>
> Some comments inline...
>
> On Fri, Dec 18, 2020 at 04:12:55PM +0100, François Ozog wrote:
> >
> >
> > On Fri, 18 Dec 2020 at 15:59, <raghu.ncstate(a)icloud.com> wrote:
> >
> >
> > Hi Francois,
> >
> >
> >
> > OPTEE should work with and without VHE enabled and be able to communicate
> > with normal world. VHE should be entirely transparent to S-EL1 partitions/
> > VM’s. Is there a reason you think VHE may not be transparent?
> >
> >
> >
> > that's what I thought when I red "With VHE, the S-EL1 exception level
> > disappears." earlier in the thread.
>
> Raghu is right that the presence or absence of VHE does not matter to OP-TEE or any S-EL1 SP. It is an S-EL2 knob.
>
> Just to be clear, "With VHE, the S-EL1 exception level disappears" is the configuration when HCR_EL2.{E2H, TGE} == {1, 1}.
>
> In this configuration, the S-EL0 SPs are controlled by the SPMC just like Host Apps are controlled by the Host OS in the link from Raghu. OP-TEE or any S-EL1 SP cannot run in this setup.
>
> The key point from the spec's perspective is that the SPMC manages the Secure
> EL2&0 translation regime. I will come back to this.
>
> The configuration in which OP-TEE runs as a VM is when HCR_EL2.{E2H, TGE} == {1, 0} or {0, 0}. The key point is that TGE=0. Whether E2H is set or not is a Hypervisor choice. It does not matter whether VHE is being used or not from the
> S-EL1 SP's perspective.
>
> In this configuration, the SPMC manages the Stage 2 translations on behalf of OP-TEE in the Secure EL1&0 translation regime,
>
> Now if a S-EL0 SP is run in this configuration, irrespective of how TGE is configured, the SPMC would have to manage both the Stage 1 and 2 translations on behalf of the S-EL0 SP in the Secure EL1&0 translation regime,
>
> Going back to the key point above, in the earlier configuration (HCR_EL2.{E2H, TGE} == {1, 1}), the SPMC would have to manage only the Stage 1 translations in Secure EL2&0 translation regime.
>
> The spec is trying to highlight this difference. Management of a S-EL0 SP in the earlier configuration is expected to be easier. Also, this is the closest we get to the FF-A configuration where the SPMC in EL3 manages S-EL0 SPs directly. The SPMC manages the Stage 1 translations in Secure EL1&0 translation regime in this case.
>
> In short, in Section 2.2.1,
>
> - In bullet 1, the spec has HCR_EL2.{E2H, TGE} == {1, 1} in mind.
>
> - In bullet 2, the spec has HCR_EL2.TGE == 0 in mind.
>
> Does this seem reasonable?
>
> >
> >
> >
> > The third picture in the link below helped me visualize the system with
> > VHE:
> >
> > https://developer.arm.com/architectures/learn-the-architecture/
> > aarch64-virtualization/virtualization-host-extensions
> >
> >
> >
> > Hypervisor/Host OS is Hafnium, Guest OS will be OPTEE, Guest Apps with be
> > TA’s. Host App’s with be S-EL0 partitions.
>
> Completely agree.
>
> cheers,
> Achin
>
> >
> >
> >
> > Thanks
> >
> > Raghu
> >
> >
> >
> > From: François Ozog <francois.ozog(a)linaro.org>
> > Sent: Friday, December 18, 2020 4:03 AM
> > To: Achin Gupta <achin.gupta(a)arm.com>
> > Cc: raghu.ncstate(a)icloud.com; hafnium(a)lists.trustedfirmware.org; Mayur
> > Gudmeti <mgudmeti(a)nvidia.com>; Ruchika Gupta <ruchika.gupta(a)linaro.org>;
> > Joakim Bech <joakim.bech(a)linaro.org>
> > Subject: Re: [Hafnium] VHE support
> >
> >
> >
> >
> >
> >
> >
> > On Thu, 17 Dec 2020 at 17:44, Achin Gupta via Hafnium <
> > hafnium(a)lists.trustedfirmware.org> wrote:
> >
> > Hi Raghu,
> >
> > On Tue, Dec 15, 2020 at 09:41:05AM -0800, raghu.ncstate(a)icloud.com
> > wrote:
> > > Hi Olivier,
> > >
> >
> > [snip]
> >
> > >
> > > Hi Achin,
> > >
> > > Question for you. I interpreted the FF-A 1.0 spec as "requiring" VHE
> > for
> > > S-EL0 partitions, based on options 1 and 2 in section 2.2.1, since
> > option 2
> > > does not mention S-EL0 partitions. Can you confirm this is the case?
> > As I
> > > was thinking through this, it seems like we might be able to pull off
> > EL0
> > > partitions without using VHE, i.e by using HCR_EL2.E2H = 0 and
> > > HCR_EL2.TGE=1.
> > > HCR_EL2.TGE is present even on ARMv8.0 architecture and removes
> > dependence
> > > on VHE, and effectively removes the necessity for this patch series.
> > > However, I'd like to understand your thoughts on why VHE is suggested
> > in the
> > > spec and if hafnium should use VHE for S-EL0 partitions or if hafnium
> > can
> > > use the approach suggested above.
> >
> > The intent in the spec is not to mandate VHE for S-EL0 SPs. We can add
> > a
> > clarification if that helps.
> >
> > The intent in the spec is to highlight that VHE is applicable to S-EL0
> > SPs only.
> >
> > With VHE, the S-EL1 exception level disappears. The SPMC can only have
> > awareness
> > of S-EL0 SPs.
> >
> > Without WHE, the S-EL1 exception level is present. As you state above,
> > there are
> > knobs in the architecture that reduce the role of S-EL1. But based upon
> > my
> > current understanding, the SPMC will see S-EL1 as the next lower
> > exception
> > level.
> >
> > If we want OPTEE to be running a number of Apps in one Secure world VM, we
> > wouldn't need WHE right?
> >
> > Can you confirm that without enabling VHE, an OPTEE VM, and its containing
> > apps, is possible and can communicate with normal world VMs?
> >
> >
> >
> > A S-EL0 SP can be "wrapped" as a S-EL1 SP such that the role of S-EL1
> > is
> > reduced. This could be done in SW (e.g. a shim layer in S-EL1), HW
> > (e.g. the TGE
> > bit above) or both. But architecturally, the SPMC will still see S-EL1.
> >
> > I hope this clarifies. That said, I do need to double check the
> > original intent
> > and impact of TGE. So please do correct me if I have misinterpreted
> > anything.
> >
> > cheers,
> > Achin
> >
> > >
> > >
> > > Thanks
> > > Raghu
> > >
> > >
> > > -----Original Message-----
> > > From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> > > Sent: Tuesday, December 15, 2020 12:18 AM
> > > To: hafnium(a)lists.trustedfirmware.org; raghu.ncstate(a)icloud.com;
> > Olivier
> > > Deprez <Olivier.Deprez(a)arm.com>
> > > Subject: Re: [Hafnium] VHE support
> > >
> > > Hi Raghu,
> > >
> > > one more
> > >
> > > 5/ maybe answer to 2/3/4 is that it requires an EL1-shim embedded
> > into
> > > Hafnium which itself ERETs to a S-EL0 partition?
> > >
> > > BTW notice my questions are obviously oriented towards the secure
> > > implementation.
> > >
> > > Regards,
> > > Olivier.
> > >
> > > ________________________________________
> > > From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf
> > of
> > > Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
> > > Sent: 15 December 2020 09:10
> > > To: hafnium(a)lists.trustedfirmware.org; raghu.ncstate(a)icloud.com
> > > Subject: Re: [Hafnium] VHE support
> > >
> > > Hi Raghu,
> > >
> > > Thanks for sharing this work.
> > >
> > > Few thoughts...
> > >
> > > 1/ I guess it requires an additional change in project/reference to
> > > eventually enable the feature for a platform (enable_vhe=1 in the gn
> > build
> > > flow)?
> > >
> > > Sorry if my questions below sound obvious, I may miss bits of the VHE
> > > architecture.
> > >
> > > 2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when
> > > Hafnium ERETs resuming a SP, this happens with the secure EL1&0
> > translation
> > > regime.
> > > Though what's the next step forward?
> > > Does this require a host OS in secure world? (like linux does when
> > booting
> > > under a Hypervisor with VHE enabled?) Would this be a VHE-enabled
> > TOS?
> > >
> > > 3/ Is the later goal to enable the secure EL2&0 translation regime?
> > >
> > > 4/ Is there anything to do with HCR_EL2.TGE?
> > >
> > > Regards,
> > > Olivier.
> > >
> > >
> > > ________________________________________
> > > From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf
> > of Raghu
> > > Krishnamurthy via Hafnium <hafnium(a)lists.trustedfirmware.org>
> > > Sent: 15 December 2020 04:57
> > > To: hafnium(a)lists.trustedfirmware.org
> > > Subject: [Hafnium] VHE support
> > >
> > > Hi All,
> > >
> > >
> > >
> > > I have a series of patches pushed to Gerrit at
> > > https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599 with
> > topic
> > > "vhe_enable". The goal of this patch series is to enable VM's in both
> > secure
> > > and normal world to run with VHE enabled(hcr_el2.e2h=1), without
> > breaking
> > > any existing functionality. This is expected to be the first step in
> > the
> > > long term goal of enabling S-EL0 partitions(and optionally EL0
> > partitions),
> > > that require VHE support, per the FF-A 1.0 Spec. I'd appreciate
> > feedback on
> > > the patches and approach taken to nominally enabling VHE. Note that
> > the FF-A
> > > 1.0 spec(AFAIK) does not expect VHE support in the normal world but
> > this
> > > patch series enables it anyway due to the wealth of available tests
> > in the
> > > hafnium test suite to help with providing confidence in the
> > implementation.
> > >
> > >
> > >
> > > The patch series has been tested as follows:
> > >
> > >
> > >
> > > Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE,
> > since
> > > the prebuilt QEMU does not support VHE.
> > >
> > > Hafnium tests using QEMU(5.2-RC4, built from source) - With and
> > without VHE,
> > > this version of QEMU supports VHE.
> > >
> > > Hafnium tests using FVP 11.12.28 - With and Without VHE.
> > >
> > > TFTF tests for secure hafnium using FVP 11.12.28 - With and without
> > VHE.
> > >
> > >
> > >
> > >
> > >
> > > Thanks
> > >
> > > Raghu
> > >
> > > --
> > > Hafnium mailing list
> > > Hafnium(a)lists.trustedfirmware.org
> > > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
> > > --
> > > Hafnium mailing list
> > > Hafnium(a)lists.trustedfirmware.org
> > > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
> > >
> > --
> > Hafnium mailing list
> > Hafnium(a)lists.trustedfirmware.org
> > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
> >
> >
> >
> >
> > --
> >
> > [uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing
> > Group
> >
> > T: +33.67221.6485
> > francois.ozog(a)linaro.org | Skype: ffozog
> >
> >
> >
> >
> >
> >
> > --
> > [uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing Group
> > T: +33.67221.6485
> > francois.ozog(a)linaro.org | Skype: ffozog
> >
>
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
On Thu, 17 Dec 2020 at 17:44, Achin Gupta via Hafnium <
hafnium(a)lists.trustedfirmware.org> wrote:
> Hi Raghu,
>
> On Tue, Dec 15, 2020 at 09:41:05AM -0800, raghu.ncstate(a)icloud.com wrote:
> > Hi Olivier,
> >
>
> [snip]
>
> >
> > Hi Achin,
> >
> > Question for you. I interpreted the FF-A 1.0 spec as "requiring" VHE for
> > S-EL0 partitions, based on options 1 and 2 in section 2.2.1, since
> option 2
> > does not mention S-EL0 partitions. Can you confirm this is the case? As I
> > was thinking through this, it seems like we might be able to pull off EL0
> > partitions without using VHE, i.e by using HCR_EL2.E2H = 0 and
> > HCR_EL2.TGE=1.
> > HCR_EL2.TGE is present even on ARMv8.0 architecture and removes
> dependence
> > on VHE, and effectively removes the necessity for this patch series.
> > However, I'd like to understand your thoughts on why VHE is suggested in
> the
> > spec and if hafnium should use VHE for S-EL0 partitions or if hafnium can
> > use the approach suggested above.
>
> The intent in the spec is not to mandate VHE for S-EL0 SPs. We can add a
> clarification if that helps.
>
> The intent in the spec is to highlight that VHE is applicable to S-EL0 SPs
> only.
>
> With VHE, the S-EL1 exception level disappears. The SPMC can only have
> awareness
> of S-EL0 SPs.
>
> Without WHE, the S-EL1 exception level is present. As you state above,
> there are
> knobs in the architecture that reduce the role of S-EL1. But based upon my
> current understanding, the SPMC will see S-EL1 as the next lower exception
> level.
>
> If we want OPTEE to be running a number of Apps in one Secure world VM, we
wouldn't need WHE right?
Can you confirm that without enabling VHE, an OPTEE VM, and its containing
apps, is possible and can communicate with normal world VMs?
A S-EL0 SP can be "wrapped" as a S-EL1 SP such that the role of S-EL1 is
> reduced. This could be done in SW (e.g. a shim layer in S-EL1), HW (e.g.
> the TGE
> bit above) or both. But architecturally, the SPMC will still see S-EL1.
>
> I hope this clarifies. That said, I do need to double check the original
> intent
> and impact of TGE. So please do correct me if I have misinterpreted
> anything.
>
> cheers,
> Achin
>
> >
> >
> > Thanks
> > Raghu
> >
> >
> > -----Original Message-----
> > From: Olivier Deprez <Olivier.Deprez(a)arm.com>
> > Sent: Tuesday, December 15, 2020 12:18 AM
> > To: hafnium(a)lists.trustedfirmware.org; raghu.ncstate(a)icloud.com; Olivier
> > Deprez <Olivier.Deprez(a)arm.com>
> > Subject: Re: [Hafnium] VHE support
> >
> > Hi Raghu,
> >
> > one more
> >
> > 5/ maybe answer to 2/3/4 is that it requires an EL1-shim embedded into
> > Hafnium which itself ERETs to a S-EL0 partition?
> >
> > BTW notice my questions are obviously oriented towards the secure
> > implementation.
> >
> > Regards,
> > Olivier.
> >
> > ________________________________________
> > From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of
> > Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
> > Sent: 15 December 2020 09:10
> > To: hafnium(a)lists.trustedfirmware.org; raghu.ncstate(a)icloud.com
> > Subject: Re: [Hafnium] VHE support
> >
> > Hi Raghu,
> >
> > Thanks for sharing this work.
> >
> > Few thoughts...
> >
> > 1/ I guess it requires an additional change in project/reference to
> > eventually enable the feature for a platform (enable_vhe=1 in the gn
> build
> > flow)?
> >
> > Sorry if my questions below sound obvious, I may miss bits of the VHE
> > architecture.
> >
> > 2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when
> > Hafnium ERETs resuming a SP, this happens with the secure EL1&0
> translation
> > regime.
> > Though what's the next step forward?
> > Does this require a host OS in secure world? (like linux does when
> booting
> > under a Hypervisor with VHE enabled?) Would this be a VHE-enabled TOS?
> >
> > 3/ Is the later goal to enable the secure EL2&0 translation regime?
> >
> > 4/ Is there anything to do with HCR_EL2.TGE?
> >
> > Regards,
> > Olivier.
> >
> >
> > ________________________________________
> > From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of
> Raghu
> > Krishnamurthy via Hafnium <hafnium(a)lists.trustedfirmware.org>
> > Sent: 15 December 2020 04:57
> > To: hafnium(a)lists.trustedfirmware.org
> > Subject: [Hafnium] VHE support
> >
> > Hi All,
> >
> >
> >
> > I have a series of patches pushed to Gerrit at
> > https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599 with topic
> > "vhe_enable". The goal of this patch series is to enable VM's in both
> secure
> > and normal world to run with VHE enabled(hcr_el2.e2h=1), without breaking
> > any existing functionality. This is expected to be the first step in the
> > long term goal of enabling S-EL0 partitions(and optionally EL0
> partitions),
> > that require VHE support, per the FF-A 1.0 Spec. I'd appreciate feedback
> on
> > the patches and approach taken to nominally enabling VHE. Note that the
> FF-A
> > 1.0 spec(AFAIK) does not expect VHE support in the normal world but this
> > patch series enables it anyway due to the wealth of available tests in
> the
> > hafnium test suite to help with providing confidence in the
> implementation.
> >
> >
> >
> > The patch series has been tested as follows:
> >
> >
> >
> > Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE,
> since
> > the prebuilt QEMU does not support VHE.
> >
> > Hafnium tests using QEMU(5.2-RC4, built from source) - With and without
> VHE,
> > this version of QEMU supports VHE.
> >
> > Hafnium tests using FVP 11.12.28 - With and Without VHE.
> >
> > TFTF tests for secure hafnium using FVP 11.12.28 - With and without VHE.
> >
> >
> >
> >
> >
> > Thanks
> >
> > Raghu
> >
> > --
> > Hafnium mailing list
> > Hafnium(a)lists.trustedfirmware.org
> > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
> > --
> > Hafnium mailing list
> > Hafnium(a)lists.trustedfirmware.org
> > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
> >
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
--
François-Frédéric Ozog | *Director Linaro Edge & Fog Computing Group*
T: +33.67221.6485
francois.ozog(a)linaro.org | Skype: ffozog
Hi Raghu,
Thanks for sharing this work.
Few thoughts...
1/ I guess it requires an additional change in project/reference to eventually enable the feature for a platform (enable_vhe=1 in the gn build flow)?
Sorry if my questions below sound obvious, I may miss bits of the VHE architecture.
2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when Hafnium ERETs resuming a SP, this happens with the secure EL1&0 translation regime.
Though what's the next step forward?
Does this require a host OS in secure world? (like linux does when booting under a Hypervisor with VHE enabled?) Would this be a VHE-enabled TOS?
3/ Is the later goal to enable the secure EL2&0 translation regime?
4/ Is there anything to do with HCR_EL2.TGE?
Regards,
Olivier.
________________________________________
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Raghu Krishnamurthy via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 15 December 2020 04:57
To: hafnium(a)lists.trustedfirmware.org
Subject: [Hafnium] VHE support
Hi All,
I have a series of patches pushed to Gerrit at
https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599 with topic
"vhe_enable". The goal of this patch series is to enable VM's in both secure
and normal world to run with VHE enabled(hcr_el2.e2h=1), without breaking
any existing functionality. This is expected to be the first step in the
long term goal of enabling S-EL0 partitions(and optionally EL0 partitions),
that require VHE support, per the FF-A 1.0 Spec. I'd appreciate feedback on
the patches and approach taken to nominally enabling VHE. Note that the FF-A
1.0 spec(AFAIK) does not expect VHE support in the normal world but this
patch series enables it anyway due to the wealth of available tests in the
hafnium test suite to help with providing confidence in the implementation.
The patch series has been tested as follows:
Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE, since
the prebuilt QEMU does not support VHE.
Hafnium tests using QEMU(5.2-RC4, built from source) - With and without VHE,
this version of QEMU supports VHE.
Hafnium tests using FVP 11.12.28 - With and Without VHE.
TFTF tests for secure hafnium using FVP 11.12.28 - With and without VHE.
Thanks
Raghu
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
Hi All,
I have a series of patches pushed to Gerrit at
https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599 with topic
"vhe_enable". The goal of this patch series is to enable VM's in both secure
and normal world to run with VHE enabled(hcr_el2.e2h=1), without breaking
any existing functionality. This is expected to be the first step in the
long term goal of enabling S-EL0 partitions(and optionally EL0 partitions),
that require VHE support, per the FF-A 1.0 Spec. I'd appreciate feedback on
the patches and approach taken to nominally enabling VHE. Note that the FF-A
1.0 spec(AFAIK) does not expect VHE support in the normal world but this
patch series enables it anyway due to the wealth of available tests in the
hafnium test suite to help with providing confidence in the implementation.
The patch series has been tested as follows:
Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE, since
the prebuilt QEMU does not support VHE.
Hafnium tests using QEMU(5.2-RC4, built from source) - With and without VHE,
this version of QEMU supports VHE.
Hafnium tests using FVP 11.12.28 - With and Without VHE.
TFTF tests for secure hafnium using FVP 11.12.28 - With and without VHE.
Thanks
Raghu
Fwd to Hafnium mailing list.
Joanna
From: TF-A <tf-a-bounces(a)lists.trustedfirmware.org> on behalf of Joanna Farley via TF-A <tf-a(a)lists.trustedfirmware.org>
Reply to: Joanna Farley <Joanna.Farley(a)arm.com>
Date: Monday, 14 December 2020 at 10:03
To: "tf-a(a)lists.trustedfirmware.org" <tf-a(a)lists.trustedfirmware.org>, "op-tee(a)lists.trustedfirmware.org" <op-tee(a)lists.trustedfirmware.org>
Subject: [TF-A] TF-A Tech Forum Agenda @ Thr 17th December 2020 16:00-1700 GMT: Introduction to the Trusted Services project
Hi All,
The next TF-A Tech Forum is scheduled for Thu 17th December 2020 16:00 – 17:00 (GMT).
As well as being posted to the TF-A mailing list this has been cross posted to OPTEE mailing list. For OPTEE attendees the Zoom call details are included below.
Agenda:
* An introduction to the Trusted Services project
* Presented by Julian Hall
* Summary
* The Trusted Services project is a new trustedfirmware.org project that provides a home for security related service components that can run in the different isolated processing environments available on Arm Cortex-A. The project attempts to promote reuse and standardization to enable a consistent set of services to be provided by firmware, independent of which isolation technology is used.
If TF-A contributors have anything they wish to present at any future TF-A tech forum please contact me to have that scheduled.
Previous sessions, both recording and presentation material can be found on the trustedfirmware.org TF-A Technical meeting webpage: https://www.trustedfirmware.org/meetings/tf-a-technical-forum/
A scheduling tracking page is also available to help track sessions suggested and being prepared: https://developer.trustedfirmware.org/w/tf_a/tf-a-tech-forum-scheduling/ Final decisions on what will be presented will be shared a few days before the next meeting and shared on the TF-A mailing list.
This is the last TF-A Tech Forum session until January 2021.
Join Zoom Meeting
https://zoom.us/j/9159704974<https://www.google.com/url?q=https%3A%2F%2Fzoom.us%2Fj%2F9159704974&sa=D&us…>
Meeting ID: 915 970 4974
One tap mobile
+16465588656,,9159704974# US (New York)
+16699009128,,9159704974# US (San Jose)
Dial by your location
+1 646 558 8656 US (New York)
+1 669 900 9128 US (San Jose)
877 853 5247 US Toll-free
888 788 0099 US Toll-free
Meeting ID: 915 970 4974
Find your local number: https://zoom.us/u/ad27hc6t7h<https://www.google.com/url?q=https%3A%2F%2Fzoom.us%2Fu%2Fad27hc6t7h&sa=D&us…>
Thanks
Joanna
Hi Raghu,
Are you referring to the gcc build flavor with which FVP is built (Linux64_GCC-6.4)?
FVP version should look more like 11.12.38 if you do:
./FVP_Base_RevC-2xAEMv8A --version
Thus far Hypervisor has been validated using qemu through the Hafnium builder (using the kokoro/build.sh script):
https://ci.trustedfirmware.org/job/hafnium-build-test-review/
Even though the build script offers the capability, there is currently no automation with FVP for the Hypervisor test suite.
We've used FVP for the secure-world / SPMC validation, but that's a different test suite within TF-A CI.
In other words, it's possible you find problems running FVP on the Hypervisor test suite, and that's worth fixing.
Regards,
Olivier.
________________________________________
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Raghu Krishnamurthy via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 08 December 2020 17:20
To: hafnium(a)lists.trustedfirmware.org
Subject: [Hafnium] Memory sharing test failures with FVP
Hi All,
I'm seeing test failures when I use FVP and run the memory_share test
suite(in primary_with_secondaries_test) on the 6.4 version of AEM FVP(works
on qemu, don't know if it works on AEM FVP 4.9). Is anybody else aware of
these failures and working on fixing them? Appears to be alignment faults.
Test logs for some of the tests show a stage 2 fault at 0x200 but that is
really because service_vm3 does not install an exception handler in EL1 and
when the data abort is taken in EL1, VBAR_EL1 is not set up, and hafnium
gets a stage 2 fault, and injects unknown exception to EL1, which faults
again in hafnium again and so on.
Easy to repro:
Update test/hftest/hftest.py to use the 6.4 version of FVP and run:
timeout --foreground 300s ./test/hftest/hftest.py --driver=fvp --out
out/reference/aem_v8a_fvp_clang --out_initrd
out/reference/aem_v8a_fvp_vm_clang --tfa --log out/reference/kokoro_log
hafnium --initrd
test/vmapi/primary_with_secondaries/primary_with_secondaries_test --suite
memory_sharing
Thanks
Raghu
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
Hi All,
I'm seeing test failures when I use FVP and run the memory_share test
suite(in primary_with_secondaries_test) on the 6.4 version of AEM FVP(works
on qemu, don't know if it works on AEM FVP 4.9). Is anybody else aware of
these failures and working on fixing them? Appears to be alignment faults.
Test logs for some of the tests show a stage 2 fault at 0x200 but that is
really because service_vm3 does not install an exception handler in EL1 and
when the data abort is taken in EL1, VBAR_EL1 is not set up, and hafnium
gets a stage 2 fault, and injects unknown exception to EL1, which faults
again in hafnium again and so on.
Easy to repro:
Update test/hftest/hftest.py to use the 6.4 version of FVP and run:
timeout --foreground 300s ./test/hftest/hftest.py --driver=fvp --out
out/reference/aem_v8a_fvp_clang --out_initrd
out/reference/aem_v8a_fvp_vm_clang --tfa --log out/reference/kokoro_log
hafnium --initrd
test/vmapi/primary_with_secondaries/primary_with_secondaries_test --suite
memory_sharing
Thanks
Raghu
Hi,
I noticed OP-TEE accesses CNTPCT_EL1 aka the physical system counter for different reasons (small delay loops, user session time, debug/trace,...)
I wonder how this should be supported when OP-TEE runs as a Secure Partition (and more generally for other TEEs and bare metal SPs) under the SPMC.
I guess this is ok as long as SPs are all statically loaded at boot time (that is all SPs use the same t0 reference time), and thus the virtual system time is similar to the physical system time.
For a VM running under Hafnium, EL1 physical counter and timer accesses are:
-not trapped when accessed from the PVM.
-trapped when accessed from secondary VMs.
(https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/h…)
Hafnium (or the SPMC):
-does not access the EL1 physical counter (CNTPCT), nor the EL1 physical timer (CNTP)
-does not setup the virtual counter offset (CNTVOFF_EL2=0 which leads to CNTVCT=CNTPCT)
-emulates the EL1 timer using virtual timer registers (CNTVCT and CNTV), and Hypervisor timer (CNTHP) for the primary VM.
Considering OP-TEE currently uses the physical counter (and does not use any architected timer), options are:
-do not trap CNTPCT accesses. The problem I see here in the longer term, the virtual time does no longer match physical time (e.g. consider dynamic loading of SPs).
-the SPMC traps CNTPCT accesses for all SPs (and maybe emulate like non-VHE KVM does, but this incurs latency in returning the virtual counter value).
-OP-TEE replaces its CNTPCT usage by CNTVCT. The latter behaves such that if virtualization is disabled accesses are made with CNTVCT=CNTPCT. I tend to think that's the approach used by linux (needs further check though).
Let me know your thoughts (@Achin, @Andrew, @JensW, others ...)
Other considerations:
-With S-EL2 enabled, the EL3 physical timer cannot be accessed from lower ELs whichever SCR_EL3.ST (the behavior is slightly different when virtualization is disabled).
-S-EL2 physical (CNTHPS) and virtual (CNTHVS) timers are unused.
-The EL1 virtual timer interrupt can fire only under the condition that the SP is voluntarily scheduled either by a direct message request or ffa_run. In other words such interrupt is never triggered by HW, but because the SPMC injects it to an SP vCPU which explicitly enabled it, when the virtual timer expires.
- I noticed CNTPCT/CNTVCT direct reads within OP-TEE and Hafnium code bases. Arm ARM quite clearly mentions that such reads must be preceded by an ISB to forbid re-ordering. I think it's worth doing the change.
Regards,
Olivier.
