On Fri, 18 Dec 2020 at 15:59, raghu.ncstate@icloud.com wrote:

Hi Francois,

OPTEE should work with and without VHE enabled and be able to communicate with normal world. VHE should be entirely transparent to S-EL1 partitions/VM’s. Is there a reason you think VHE may not be transparent?

that's what I thought when I red "With VHE, the S-EL1 exception level disappears." earlier in the thread.

The third picture in the link below helped me visualize the system with VHE:

https://developer.arm.com/architectures/learn-the-architecture/aarch64-virtu...

Hypervisor/Host OS is Hafnium, Guest OS will be OPTEE, Guest Apps with be TA’s. Host App’s with be S-EL0 partitions.

Thanks

Raghu

*From:* François Ozog francois.ozog@linaro.org *Sent:* Friday, December 18, 2020 4:03 AM *To:* Achin Gupta achin.gupta@arm.com *Cc:* raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org; Mayur Gudmeti mgudmeti@nvidia.com; Ruchika Gupta ruchika.gupta@linaro.org; Joakim Bech joakim.bech@linaro.org *Subject:* Re: [Hafnium] VHE support

On Thu, 17 Dec 2020 at 17:44, Achin Gupta via Hafnium < hafnium@lists.trustedfirmware.org> wrote:

Hi Raghu,

On Tue, Dec 15, 2020 at 09:41:05AM -0800, raghu.ncstate@icloud.com wrote:

...

Hi Olivier,

[snip]

...

Hi Achin,

Question for you. I interpreted the FF-A 1.0 spec as "requiring" VHE for S-EL0 partitions, based on options 1 and 2 in section 2.2.1, since

option 2

...

does not mention S-EL0 partitions. Can you confirm this is the case? As I was thinking through this, it seems like we might be able to pull off EL0 partitions without using VHE, i.e by using HCR_EL2.E2H = 0 and HCR_EL2.TGE=1. HCR_EL2.TGE is present even on ARMv8.0 architecture and removes

dependence

...

on VHE, and effectively removes the necessity for this patch series. However, I'd like to understand your thoughts on why VHE is suggested in

the

...

spec and if hafnium should use VHE for S-EL0 partitions or if hafnium can use the approach suggested above.

The intent in the spec is not to mandate VHE for S-EL0 SPs. We can add a clarification if that helps.

The intent in the spec is to highlight that VHE is applicable to S-EL0 SPs only.

With VHE, the S-EL1 exception level disappears. The SPMC can only have awareness of S-EL0 SPs.

Without WHE, the S-EL1 exception level is present. As you state above, there are knobs in the architecture that reduce the role of S-EL1. But based upon my current understanding, the SPMC will see S-EL1 as the next lower exception level.

If we want OPTEE to be running a number of Apps in one Secure world VM, we wouldn't need WHE right?

Can you confirm that without enabling VHE, an OPTEE VM, and its containing apps, is possible and can communicate with normal world VMs?

A S-EL0 SP can be "wrapped" as a S-EL1 SP such that the role of S-EL1 is reduced. This could be done in SW (e.g. a shim layer in S-EL1), HW (e.g. the TGE bit above) or both. But architecturally, the SPMC will still see S-EL1.

I hope this clarifies. That said, I do need to double check the original intent and impact of TGE. So please do correct me if I have misinterpreted anything.

cheers, Achin

...

Thanks Raghu

-----Original Message----- From: Olivier Deprez Olivier.Deprez@arm.com Sent: Tuesday, December 15, 2020 12:18 AM To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com; Olivier Deprez Olivier.Deprez@arm.com Subject: Re: [Hafnium] VHE support

Hi Raghu,

one more

5/ maybe answer to 2/3/4 is that it requires an EL1-shim embedded into Hafnium which itself ERETs to a S-EL0 partition?

BTW notice my questions are obviously oriented towards the secure implementation.

Regards, Olivier.

---

From: Hafnium hafnium-bounces@lists.trustedfirmware.org on behalf of Olivier Deprez via Hafnium hafnium@lists.trustedfirmware.org Sent: 15 December 2020 09:10 To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com Subject: Re: [Hafnium] VHE support

Hi Raghu,

Thanks for sharing this work.

Few thoughts...

1/ I guess it requires an additional change in project/reference to eventually enable the feature for a platform (enable_vhe=1 in the gn

build

...

flow)?

Sorry if my questions below sound obvious, I may miss bits of the VHE architecture.

2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when Hafnium ERETs resuming a SP, this happens with the secure EL1&0

translation

...

regime. Though what's the next step forward? Does this require a host OS in secure world? (like linux does when

booting

...

under a Hypervisor with VHE enabled?) Would this be a VHE-enabled TOS?

3/ Is the later goal to enable the secure EL2&0 translation regime?

4/ Is there anything to do with HCR_EL2.TGE?

Regards, Olivier.

---

From: Hafnium hafnium-bounces@lists.trustedfirmware.org on behalf of

Raghu

...

Krishnamurthy via Hafnium hafnium@lists.trustedfirmware.org Sent: 15 December 2020 04:57 To: hafnium@lists.trustedfirmware.org Subject: [Hafnium] VHE support

Hi All,

I have a series of patches pushed to Gerrit at https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599 with topic "vhe_enable". The goal of this patch series is to enable VM's in both

secure

...

and normal world to run with VHE enabled(hcr_el2.e2h=1), without breaking any existing functionality. This is expected to be the first step in the long term goal of enabling S-EL0 partitions(and optionally EL0

partitions),

...

that require VHE support, per the FF-A 1.0 Spec. I'd appreciate feedback

on

...

the patches and approach taken to nominally enabling VHE. Note that the

FF-A

...

1.0 spec(AFAIK) does not expect VHE support in the normal world but this patch series enables it anyway due to the wealth of available tests in

the

...

hafnium test suite to help with providing confidence in the

implementation.

...

The patch series has been tested as follows:

Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE,

since

...

the prebuilt QEMU does not support VHE.

Hafnium tests using QEMU(5.2-RC4, built from source) - With and without

VHE,

...

this version of QEMU supports VHE.

Hafnium tests using FVP 11.12.28 - With and Without VHE.

TFTF tests for secure hafnium using FVP 11.12.28 - With and without VHE.

Thanks

Raghu

-- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium -- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium

-- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium

--

*François-Frédéric Ozog* | *Director Linaro Edge & Fog Computing Group*

T: +33.67221.6485 francois.ozog@linaro.org | Skype: ffozog

Thanks Achin.

...

Now if a S-EL0 SP is run in this configuration, irrespective of how TGE is configured, the SPMC would have to manage both the Stage 1 and 2 translations on behalf of the S-EL0 SP in the Secure EL1&0 translation regime,

[RK] Not sure I understand the statement "irrespective of how TGE is configured". If HCR_EL2.E2H,TGE=1,1, then SPMC would manage Secure EL2&0 translation regime for the S-EL0 SP and crucially, no stage 2 translations. Your statement applies only when HCR_EL2.{E2H,TGE} = {1,1}. The statement right after this contradicts "irrespective of how TGE is configured".

...

Management of a S-EL0 SP in the earlier configuration is expected to be easier.

[RK] Sorry, which configuration? Earlier is vague.

...

Does this seem reasonable?

Yes, this is reasonable. However, what configuration of HCR_EL2.{E2H, TGE} should hafnium use to manage S-EL0 SP's? Should it use {0, 1} (not in spec) or {1, 1}( Section 2.2.1, Bullet 1)?

Thanks Raghu

-----Original Message----- From: Achin Gupta achin.gupta@arm.com Sent: Friday, December 18, 2020 1:57 PM To: François Ozog francois.ozog@linaro.org; raghu.ncstate@icloud.com Cc: raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org; Mayur Gudmeti mgudmeti@nvidia.com; Ruchika Gupta ruchika.gupta@linaro.org; Joakim Bech joakim.bech@linaro.org Subject: Re: [Hafnium] VHE support

Hi Francois and Raghu,

Some comments inline...

On Fri, Dec 18, 2020 at 04:12:55PM +0100, François Ozog wrote:

On Fri, 18 Dec 2020 at 15:59, raghu.ncstate@icloud.com wrote:

Hi Francois,



OPTEE should work with and without VHE enabled and be able to communicate
with normal world. VHE should be entirely transparent to S-EL1 partitions/
VM’s. Is there a reason you think VHE may not be transparent?

that's what I thought when I red "With VHE, the S-EL1 exception level disappears." earlier in the thread.

Raghu is right that the presence or absence of VHE does not matter to OP-TEE or any S-EL1 SP. It is an S-EL2 knob.

Just to be clear, "With VHE, the S-EL1 exception level disappears" is the configuration when HCR_EL2.{E2H, TGE} == {1, 1}.

In this configuration, the S-EL0 SPs are controlled by the SPMC just like Host Apps are controlled by the Host OS in the link from Raghu. OP-TEE or any S-EL1 SP cannot run in this setup.

The key point from the spec's perspective is that the SPMC manages the Secure EL2&0 translation regime. I will come back to this.

The configuration in which OP-TEE runs as a VM is when HCR_EL2.{E2H, TGE} == {1, 0} or {0, 0}. The key point is that TGE=0. Whether E2H is set or not is a Hypervisor choice. It does not matter whether VHE is being used or not from the S-EL1 SP's perspective.

In this configuration, the SPMC manages the Stage 2 translations on behalf of OP-TEE in the Secure EL1&0 translation regime,

Now if a S-EL0 SP is run in this configuration, irrespective of how TGE is configured, the SPMC would have to manage both the Stage 1 and 2 translations on behalf of the S-EL0 SP in the Secure EL1&0 translation regime,

Going back to the key point above, in the earlier configuration (HCR_EL2.{E2H, TGE} == {1, 1}), the SPMC would have to manage only the Stage 1 translations in Secure EL2&0 translation regime.

The spec is trying to highlight this difference. Management of a S-EL0 SP in the earlier configuration is expected to be easier. Also, this is the closest we get to the FF-A configuration where the SPMC in EL3 manages S-EL0 SPs directly. The SPMC manages the Stage 1 translations in Secure EL1&0 translation regime in this case.

In short, in Section 2.2.1,

- In bullet 1, the spec has HCR_EL2.{E2H, TGE} == {1, 1} in mind.

- In bullet 2, the spec has HCR_EL2.TGE == 0 in mind.

Does this seem reasonable?

The third picture in the link below helped me visualize the system with
VHE:

https://developer.arm.com/architectures/learn-the-architecture/
aarch64-virtualization/virtualization-host-extensions



Hypervisor/Host OS is Hafnium, Guest OS will be OPTEE, Guest Apps with be
TA’s. Host App’s with be S-EL0 partitions.

Completely agree.

cheers, Achin

Thanks

Raghu



From: François Ozog <francois.ozog@linaro.org>
Sent: Friday, December 18, 2020 4:03 AM
To: Achin Gupta <achin.gupta@arm.com>
Cc: raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org; Mayur
Gudmeti <mgudmeti@nvidia.com>; Ruchika Gupta <ruchika.gupta@linaro.org>;
Joakim Bech <joakim.bech@linaro.org>
Subject: Re: [Hafnium] VHE support







On Thu, 17 Dec 2020 at 17:44, Achin Gupta via Hafnium <
hafnium@lists.trustedfirmware.org> wrote:

    Hi Raghu,

    On Tue, Dec 15, 2020 at 09:41:05AM -0800, raghu.ncstate@icloud.com
    wrote:
    > Hi Olivier,
    >

    [snip]

    >
    > Hi Achin,
    >
    > Question for you. I interpreted the FF-A 1.0 spec as "requiring" VHE
    for
    > S-EL0 partitions, based on options 1 and 2 in section 2.2.1, since
    option 2
    > does not mention S-EL0 partitions. Can you confirm this is the case?
    As I
    > was thinking through this, it seems like we might be able to pull off
    EL0
    > partitions without using VHE, i.e by using HCR_EL2.E2H = 0 and
    > HCR_EL2.TGE=1.
    > HCR_EL2.TGE is present even on ARMv8.0 architecture and removes
    dependence
    > on VHE, and effectively removes the necessity for this patch series.
    > However, I'd like to understand your thoughts on why VHE is suggested
    in the
    > spec and if hafnium should use VHE for S-EL0 partitions or if hafnium
    can
    > use the approach suggested above.

    The intent in the spec is not to mandate VHE for S-EL0 SPs. We can add
    a
    clarification if that helps.

    The intent in the spec is to highlight that VHE is applicable to S-EL0
    SPs only.

    With VHE, the S-EL1 exception level disappears. The SPMC can only have
    awareness
    of S-EL0 SPs.

    Without WHE, the S-EL1 exception level is present. As you state above,
    there are
    knobs in the architecture that reduce the role of S-EL1. But based upon
    my
    current understanding, the SPMC will see S-EL1 as the next lower
    exception
    level.

If we want OPTEE to be running a number of Apps in one Secure world VM, we
wouldn't need WHE right?

Can you confirm that without enabling VHE, an OPTEE VM, and its containing
apps, is possible and can communicate with  normal world VMs?



    A S-EL0 SP can be "wrapped" as a S-EL1 SP such that the role of S-EL1
    is
    reduced. This could be done in SW (e.g. a shim layer in S-EL1), HW
    (e.g. the TGE
    bit above) or both. But architecturally, the SPMC will still see S-EL1.

    I hope this clarifies. That said, I do need to double check the
    original intent
    and impact of TGE. So please do correct me if I have misinterpreted
    anything.

    cheers,
    Achin

    >
    >
    > Thanks
    > Raghu
    >
    >
    > -----Original Message-----
    > From: Olivier Deprez <Olivier.Deprez@arm.com>
    > Sent: Tuesday, December 15, 2020 12:18 AM
    > To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com;
    Olivier
    > Deprez <Olivier.Deprez@arm.com>
    > Subject: Re: [Hafnium] VHE support
    >
    > Hi Raghu,
    >
    > one more
    >
    > 5/ maybe answer to 2/3/4 is that it requires an EL1-shim embedded
    into
    > Hafnium which itself ERETs to a S-EL0 partition?
    >
    > BTW notice my questions are obviously oriented towards the secure
    > implementation.
    >
    > Regards,
    > Olivier.
    >
    > ________________________________________
    > From: Hafnium <hafnium-bounces@lists.trustedfirmware.org> on behalf
    of
    > Olivier Deprez via Hafnium <hafnium@lists.trustedfirmware.org>
    > Sent: 15 December 2020 09:10
    > To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com
    > Subject: Re: [Hafnium] VHE support
    >
    > Hi Raghu,
    >
    > Thanks for sharing this work.
    >
    > Few thoughts...
    >
    > 1/ I guess it requires an additional change in project/reference to
    > eventually enable the feature for a platform (enable_vhe=1 in the gn
    build
    > flow)?
    >
    > Sorry if my questions below sound obvious, I may miss bits of the VHE
    > architecture.
    >
    > 2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when
    > Hafnium ERETs resuming a SP, this happens with the secure EL1&0
    translation
    > regime.
    > Though what's the next step forward?
    > Does this require a host OS in secure world? (like linux does when
    booting
    > under a Hypervisor with VHE enabled?) Would this be a VHE-enabled
    TOS?
    >
    > 3/ Is the later goal to enable the secure EL2&0 translation regime?
    >
    > 4/ Is there anything to do with HCR_EL2.TGE?
    >
    > Regards,
    > Olivier.
    >
    >
    > ________________________________________
    > From: Hafnium <hafnium-bounces@lists.trustedfirmware.org> on behalf
    of Raghu
    > Krishnamurthy via Hafnium <hafnium@lists.trustedfirmware.org>
    > Sent: 15 December 2020 04:57
    > To: hafnium@lists.trustedfirmware.org
    > Subject: [Hafnium] VHE support
    >
    > Hi All,
    >
    >
    >
    > I have a series of patches pushed to Gerrit at
    > https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599  with
    topic
    > "vhe_enable". The goal of this patch series is to enable VM's in both
    secure
    > and normal world to run with VHE enabled(hcr_el2.e2h=1), without
    breaking
    > any existing functionality. This is expected to be the first step in
    the
    > long term goal of enabling S-EL0 partitions(and optionally EL0
    partitions),
    > that require VHE support, per the FF-A 1.0 Spec. I'd appreciate
    feedback on
    > the patches and approach taken to nominally enabling VHE. Note that
    the FF-A
    > 1.0 spec(AFAIK) does not expect VHE support in the normal world but
    this
    > patch series enables it anyway due to the wealth of available tests
    in the
    > hafnium test suite to help with providing confidence in the
    implementation.
    >
    >
    >
    > The patch series has been tested as follows:
    >
    >
    >
    > Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE,
    since
    > the prebuilt QEMU does not support VHE.
    >
    > Hafnium tests using QEMU(5.2-RC4, built from source) - With and
    without VHE,
    > this version of QEMU supports VHE.
    >
    > Hafnium tests using FVP 11.12.28 - With and Without VHE.
    >
    > TFTF tests for secure hafnium using FVP 11.12.28 - With and without
    VHE.
    >
    >
    >
    >
    >
    > Thanks
    >
    > Raghu
    >
    > --
    > Hafnium mailing list
    > Hafnium@lists.trustedfirmware.org
    > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
    > --
    > Hafnium mailing list
    > Hafnium@lists.trustedfirmware.org
    > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
    >
    --
    Hafnium mailing list
    Hafnium@lists.trustedfirmware.org
    https://lists.trustedfirmware.org/mailman/listinfo/hafnium




--

[uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing 

Group

     T: +33.67221.6485
     francois.ozog@linaro.org | Skype: ffozog

-- [uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing Group T: +33.67221.6485 francois.ozog@linaro.org | Skype: ffozog

Thanks Achin. Makes sense. I'll explore both options and push patches.

Francois,

The diagram looks good to me. That is the model I'm trying to enable.

Thanks Raghu

-----Original Message----- From: Achin Gupta achin.gupta@arm.com Sent: Saturday, December 19, 2020 6:23 AM To: raghu.ncstate@icloud.com Cc: 'François Ozog' francois.ozog@linaro.org; hafnium@lists.trustedfirmware.org; 'Mayur Gudmeti' mgudmeti@nvidia.com; 'Ruchika Gupta' ruchika.gupta@linaro.org; 'Joakim Bech' joakim.bech@linaro.org Subject: Re: [Hafnium] VHE support

Hi Raghu,

On Fri, Dec 18, 2020 at 02:39:44PM -0800, raghu.ncstate@icloud.com wrote:

Thanks Achin.

...

...

Now if a S-EL0 SP is run in this configuration, irrespective of how TGE is configured, the SPMC would have to manage both the Stage 1 and 2 translations on behalf of the S-EL0 SP in the Secure EL1&0 translation regime,

[RK] Not sure I understand the statement "irrespective of how TGE is configured". If HCR_EL2.E2H,TGE=1,1, then SPMC would manage Secure EL2&0 translation regime for the S-EL0 SP and crucially, no stage 2 translations. Your statement applies only when HCR_EL2.{E2H,TGE} = {1,1}. The statement right after this contradicts "irrespective of how TGE is configured".

Apologies. I meant E2H and not TGE i.e. if HCR_EL2.TGE=0, then the SPMC has to manage both the Stage 1 and 2 translations on behalf of a S-EL0 SP in the Secure EL1&0 translation regime,

...

...

Management of a S-EL0 SP in the earlier configuration is expected to be easier.

[RK] Sorry, which configuration? Earlier is vague.

HCR_EL2.{E2H,TGE} = {1,1}

...

...

Does this seem reasonable?

Yes, this is reasonable. However, what configuration of HCR_EL2.{E2H, TGE} should hafnium use to manage S-EL0 SP's? Should it use {0, 1} (not in spec) or {1, 1}( Section 2.2.1, Bullet 1)?

HCR_EL2.{E2H,TGE} = {1,1} reduces the number of translation stages managed by the SPMC for S-EL0 SPs. It allows them to be managed like apps instead of VMs which is the case pre-v8.4. I think this is the better configuration for a system that only deploys S-EL0 SPs.

The spec does not specify the values of E2H or TGE. So {0, 1} is not disallowed by the spec either.

Looks like that the choices we have for Hafnium are:

1. {1, 1} for S-EL0 SPs and {1, 0} for S-EL1 SPs 2. {0, 1} for S-EL0 SPs and {0, 0} for S-EL1 SPs

We would need to understand the impact of these choices on the Hafnium implementation before making a call.

The v1.1 spec can recommend the sensible choices to avoid confusion.

cheers, Achin

Thanks Raghu

-----Original Message----- From: Achin Gupta achin.gupta@arm.com Sent: Friday, December 18, 2020 1:57 PM To: François Ozog francois.ozog@linaro.org; raghu.ncstate@icloud.com Cc: raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org; Mayur Gudmeti mgudmeti@nvidia.com; Ruchika Gupta ruchika.gupta@linaro.org; Joakim Bech joakim.bech@linaro.org Subject: Re: [Hafnium] VHE support

Hi Francois and Raghu,

Some comments inline...

On Fri, Dec 18, 2020 at 04:12:55PM +0100, François Ozog wrote:

...

On Fri, 18 Dec 2020 at 15:59, raghu.ncstate@icloud.com wrote:

Hi Francois,



OPTEE should work with and without VHE enabled and be able to communicate
with normal world. VHE should be entirely transparent to S-EL1 partitions/
VM’s. Is there a reason you think VHE may not be transparent?

that's what I thought when I red "With VHE, the S-EL1 exception level disappears." earlier in the thread.

Raghu is right that the presence or absence of VHE does not matter to OP-TEE or any S-EL1 SP. It is an S-EL2 knob.

Just to be clear, "With VHE, the S-EL1 exception level disappears" is the configuration when HCR_EL2.{E2H, TGE} == {1, 1}.

In this configuration, the S-EL0 SPs are controlled by the SPMC just like Host Apps are controlled by the Host OS in the link from Raghu. OP-TEE or any S-EL1 SP cannot run in this setup.

The key point from the spec's perspective is that the SPMC manages the Secure EL2&0 translation regime. I will come back to this.

The configuration in which OP-TEE runs as a VM is when HCR_EL2.{E2H, TGE} == {1, 0} or {0, 0}. The key point is that TGE=0. Whether E2H is set or not is a Hypervisor choice. It does not matter whether VHE is being used or not from the S-EL1 SP's perspective.

In this configuration, the SPMC manages the Stage 2 translations on behalf of OP-TEE in the Secure EL1&0 translation regime,

Now if a S-EL0 SP is run in this configuration, irrespective of how TGE is configured, the SPMC would have to manage both the Stage 1 and 2 translations on behalf of the S-EL0 SP in the Secure EL1&0 translation regime,

Going back to the key point above, in the earlier configuration (HCR_EL2.{E2H, TGE} == {1, 1}), the SPMC would have to manage only the Stage 1 translations in Secure EL2&0 translation regime.

The spec is trying to highlight this difference. Management of a S-EL0 SP in the earlier configuration is expected to be easier. Also, this is the closest we get to the FF-A configuration where the SPMC in EL3 manages S-EL0 SPs directly. The SPMC manages the Stage 1 translations in Secure EL1&0 translation regime in this case.

In short, in Section 2.2.1,

• In bullet 1, the spec has HCR_EL2.{E2H, TGE} == {1, 1} in mind.

• In bullet 2, the spec has HCR_EL2.TGE == 0 in mind.

Does this seem reasonable?

...

The third picture in the link below helped me visualize the system with
VHE:

https://developer.arm.com/architectures/learn-the-architecture/
aarch64-virtualization/virtualization-host-extensions



Hypervisor/Host OS is Hafnium, Guest OS will be OPTEE, Guest Apps with be
TA’s. Host App’s with be S-EL0 partitions.

Completely agree.

cheers, Achin

...

Thanks

Raghu



From: François Ozog <francois.ozog@linaro.org>
Sent: Friday, December 18, 2020 4:03 AM
To: Achin Gupta <achin.gupta@arm.com>
Cc: raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org; Mayur
Gudmeti <mgudmeti@nvidia.com>; Ruchika Gupta <ruchika.gupta@linaro.org>;
Joakim Bech <joakim.bech@linaro.org>
Subject: Re: [Hafnium] VHE support







On Thu, 17 Dec 2020 at 17:44, Achin Gupta via Hafnium <
hafnium@lists.trustedfirmware.org> wrote:

    Hi Raghu,

    On Tue, Dec 15, 2020 at 09:41:05AM -0800, raghu.ncstate@icloud.com
    wrote:
    > Hi Olivier,
    >

    [snip]

    >
    > Hi Achin,
    >
    > Question for you. I interpreted the FF-A 1.0 spec as "requiring" VHE
    for
    > S-EL0 partitions, based on options 1 and 2 in section 2.2.1, since
    option 2
    > does not mention S-EL0 partitions. Can you confirm this is the case?
    As I
    > was thinking through this, it seems like we might be able to pull off
    EL0
    > partitions without using VHE, i.e by using HCR_EL2.E2H = 0 and
    > HCR_EL2.TGE=1.
    > HCR_EL2.TGE is present even on ARMv8.0 architecture and removes
    dependence
    > on VHE, and effectively removes the necessity for this patch series.
    > However, I'd like to understand your thoughts on why VHE is suggested
    in the
    > spec and if hafnium should use VHE for S-EL0 partitions or if hafnium
    can
    > use the approach suggested above.

    The intent in the spec is not to mandate VHE for S-EL0 SPs. We can add
    a
    clarification if that helps.

    The intent in the spec is to highlight that VHE is applicable to S-EL0
    SPs only.

    With VHE, the S-EL1 exception level disappears. The SPMC can only have
    awareness
    of S-EL0 SPs.

    Without WHE, the S-EL1 exception level is present. As you state above,
    there are
    knobs in the architecture that reduce the role of S-EL1. But based upon
    my
    current understanding, the SPMC will see S-EL1 as the next lower
    exception
    level.

If we want OPTEE to be running a number of Apps in one Secure world VM, we
wouldn't need WHE right?

Can you confirm that without enabling VHE, an OPTEE VM, and its containing
apps, is possible and can communicate with  normal world VMs?



    A S-EL0 SP can be "wrapped" as a S-EL1 SP such that the role of S-EL1
    is
    reduced. This could be done in SW (e.g. a shim layer in S-EL1), HW
    (e.g. the TGE
    bit above) or both. But architecturally, the SPMC will still see S-EL1.

    I hope this clarifies. That said, I do need to double check the
    original intent
    and impact of TGE. So please do correct me if I have misinterpreted
    anything.

    cheers,
    Achin

    >
    >
    > Thanks
    > Raghu
    >
    >
    > -----Original Message-----
    > From: Olivier Deprez <Olivier.Deprez@arm.com>
    > Sent: Tuesday, December 15, 2020 12:18 AM
    > To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com;
    Olivier
    > Deprez <Olivier.Deprez@arm.com>
    > Subject: Re: [Hafnium] VHE support
    >
    > Hi Raghu,
    >
    > one more
    >
    > 5/ maybe answer to 2/3/4 is that it requires an EL1-shim embedded
    into
    > Hafnium which itself ERETs to a S-EL0 partition?
    >
    > BTW notice my questions are obviously oriented towards the secure
    > implementation.
    >
    > Regards,
    > Olivier.
    >
    > ________________________________________
    > From: Hafnium <hafnium-bounces@lists.trustedfirmware.org> on behalf
    of
    > Olivier Deprez via Hafnium <hafnium@lists.trustedfirmware.org>
    > Sent: 15 December 2020 09:10
    > To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com
    > Subject: Re: [Hafnium] VHE support
    >
    > Hi Raghu,
    >
    > Thanks for sharing this work.
    >
    > Few thoughts...
    >
    > 1/ I guess it requires an additional change in project/reference to
    > eventually enable the feature for a platform (enable_vhe=1 in the gn
    build
    > flow)?
    >
    > Sorry if my questions below sound obvious, I may miss bits of the VHE
    > architecture.
    >
    > 2/ The changes are effectively toggling HCR_EL2.E2H=1. Currently when
    > Hafnium ERETs resuming a SP, this happens with the secure EL1&0
    translation
    > regime.
    > Though what's the next step forward?
    > Does this require a host OS in secure world? (like linux does when
    booting
    > under a Hypervisor with VHE enabled?) Would this be a VHE-enabled
    TOS?
    >
    > 3/ Is the later goal to enable the secure EL2&0 translation regime?
    >
    > 4/ Is there anything to do with HCR_EL2.TGE?
    >
    > Regards,
    > Olivier.
    >
    >
    > ________________________________________
    > From: Hafnium <hafnium-bounces@lists.trustedfirmware.org> on behalf
    of Raghu
    > Krishnamurthy via Hafnium <hafnium@lists.trustedfirmware.org>
    > Sent: 15 December 2020 04:57
    > To: hafnium@lists.trustedfirmware.org
    > Subject: [Hafnium] VHE support
    >
    > Hi All,
    >
    >
    >
    > I have a series of patches pushed to Gerrit at
    > https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599  with
    topic
    > "vhe_enable". The goal of this patch series is to enable VM's in both
    secure
    > and normal world to run with VHE enabled(hcr_el2.e2h=1), without
    breaking
    > any existing functionality. This is expected to be the first step in
    the
    > long term goal of enabling S-EL0 partitions(and optionally EL0
    partitions),
    > that require VHE support, per the FF-A 1.0 Spec. I'd appreciate
    feedback on
    > the patches and approach taken to nominally enabling VHE. Note that
    the FF-A
    > 1.0 spec(AFAIK) does not expect VHE support in the normal world but
    this
    > patch series enables it anyway due to the wealth of available tests
    in the
    > hafnium test suite to help with providing confidence in the
    implementation.
    >
    >
    >
    > The patch series has been tested as follows:
    >
    >
    >
    > Hafnium tests using QEMU(prebuilt in the hafnium repo) - Without VHE,
    since
    > the prebuilt QEMU does not support VHE.
    >
    > Hafnium tests using QEMU(5.2-RC4, built from source) - With and
    without VHE,
    > this version of QEMU supports VHE.
    >
    > Hafnium tests using FVP 11.12.28 - With and Without VHE.
    >
    > TFTF tests for secure hafnium using FVP 11.12.28 - With and without
    VHE.
    >
    >
    >
    >
    >
    > Thanks
    >
    > Raghu
    >
    > --
    > Hafnium mailing list
    > Hafnium@lists.trustedfirmware.org
    > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
    > --
    > Hafnium mailing list
    > Hafnium@lists.trustedfirmware.org
    > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
    >
    --
    Hafnium mailing list
    Hafnium@lists.trustedfirmware.org
    https://lists.trustedfirmware.org/mailman/listinfo/hafnium




--

[uc] François-Frédéric Ozog | Director Linaro Edge & Fog 

Computing Group

     T: +33.67221.6485
     francois.ozog@linaro.org | Skype: ffozog

-- [uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing Group T: +33.67221.6485 francois.ozog@linaro.org | Skype: ffozog

Hi Francois,

Just to complete the loop, your diagram makes sense.

cheers, Achin

On Sat, Dec 19, 2020 at 06:02:57AM +0100, François Ozog wrote:

On Fri, 18 Dec 2020 at 22:58, Achin Gupta achin.gupta@arm.com wrote:

Hi Francois and Raghu,

Some comments inline...

On Fri, Dec 18, 2020 at 04:12:55PM +0100, François Ozog wrote:
>
>
> On Fri, 18 Dec 2020 at 15:59, <raghu.ncstate@icloud.com> wrote:
>
>
>     Hi Francois,
>
>
>
>     OPTEE should work with and without VHE enabled and be able to
communicate
>     with normal world. VHE should be entirely transparent to S-EL1
partitions/
>     VM’s. Is there a reason you think VHE may not be transparent?
>
>
>
> that's what I thought when I red "With VHE, the S-EL1 exception level
> disappears." earlier in the thread.

Raghu is right that the presence or absence of VHE does not matter to
OP-TEE or
any S-EL1 SP. It is an S-EL2 knob.

Just to be clear, "With VHE, the S-EL1 exception level disappears" is the
configuration when HCR_EL2.{E2H, TGE} == {1, 1}.

In this configuration, the S-EL0 SPs are controlled by the SPMC just like
Host
Apps are controlled by the Host OS in the link from Raghu. OP-TEE or any
S-EL1
SP cannot run in this setup.

The key point from the spec's perspective is that the SPMC manages the
Secure
EL2&0 translation regime. I will come back to this.

The configuration in which OP-TEE runs as a VM is when HCR_EL2.{E2H, TGE} =
= {1,
0} or {0, 0}. The key point is that TGE=0. Whether E2H is set or not is a
Hypervisor choice. It does not matter whether VHE is being used or not from
the
S-EL1 SP's perspective.

In this configuration, the SPMC manages the Stage 2 translations on behalf
of
OP-TEE in the Secure EL1&0 translation regime,

Now if a S-EL0 SP is run in this configuration, irrespective of how TGE is
configured, the SPMC would have to manage both the Stage 1 and 2
translations on
behalf of the S-EL0 SP in the Secure EL1&0 translation regime,

Going back to the key point above, in the earlier configuration (HCR_EL2.
{E2H,
TGE} == {1, 1}), the SPMC would have to manage only the Stage 1
translations in
Secure EL2&0 translation regime.

The spec is trying to highlight this difference. Management of a S-EL0 SP
in the
earlier configuration is expected to be easier. Also, this is the closest
we get
to the FF-A configuration where the SPMC in EL3 manages S-EL0 SPs directly.
The
SPMC manages the Stage 1 translations in Secure EL1&0 translation regime in
this
case.

In short, in Section 2.2.1,

- In bullet 1, the spec has HCR_EL2.{E2H, TGE} == {1, 1} in mind.

- In bullet 2, the spec has HCR_EL2.TGE == 0 in mind.

Does this seem reasonable?

Thank you for taking the time to fully explain this. I think I understand now. To be 200% sure I made a diagram of a possible runnin situation:

• Secure Hafnium Apps are actually what we called Secure Partitions and they

can be delegated secure hardware. Those Hafnium Apps can be FF-A services or StandAloneMM.

• there can be multiple Trusted OSes and may be multiple instances of OP-TEE

(one could imagine an OP-TEE "attached" to a normal world VM and a platform OP-TEE).

>
>
>
>     The third picture in the link below helped me visualize the system
with
>     VHE:
>
>     https://developer.arm.com/architectures/learn-the-architecture/
>     aarch64-virtualization/virtualization-host-extensions
>
>
>
>     Hypervisor/Host OS is Hafnium, Guest OS will be OPTEE, Guest Apps
with be
>     TA’s. Host App’s with be S-EL0 partitions.

Completely agree.

cheers,
Achin

>
>
>
>     Thanks
>
>     Raghu
>
>
>
>     From: François Ozog <francois.ozog@linaro.org>
>     Sent: Friday, December 18, 2020 4:03 AM
>     To: Achin Gupta <achin.gupta@arm.com>
>     Cc: raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org;
Mayur
>     Gudmeti <mgudmeti@nvidia.com>; Ruchika Gupta <
ruchika.gupta@linaro.org>;
>     Joakim Bech <joakim.bech@linaro.org>
>     Subject: Re: [Hafnium] VHE support
>
>
>
>
>
>
>
>     On Thu, 17 Dec 2020 at 17:44, Achin Gupta via Hafnium <
>     hafnium@lists.trustedfirmware.org> wrote:
>
>         Hi Raghu,
>
>         On Tue, Dec 15, 2020 at 09:41:05AM -0800,
raghu.ncstate@icloud.com
>         wrote:
>         > Hi Olivier,
>         >
>
>         [snip]
>
>         >
>         > Hi Achin,
>         >
>         > Question for you. I interpreted the FF-A 1.0 spec as
"requiring" VHE
>         for
>         > S-EL0 partitions, based on options 1 and 2 in section 2.2.1,
since
>         option 2
>         > does not mention S-EL0 partitions. Can you confirm this is the
case?
>         As I
>         > was thinking through this, it seems like we might be able to
pull off
>         EL0
>         > partitions without using VHE, i.e by using HCR_EL2.E2H = 0 and
>         > HCR_EL2.TGE=1.
>         > HCR_EL2.TGE is present even on ARMv8.0 architecture and removes
>         dependence
>         > on VHE, and effectively removes the necessity for this patch
series.
>         > However, I'd like to understand your thoughts on why VHE is
suggested
>         in the
>         > spec and if hafnium should use VHE for S-EL0 partitions or if
hafnium
>         can
>         > use the approach suggested above.
>
>         The intent in the spec is not to mandate VHE for S-EL0 SPs. We
can add
>         a
>         clarification if that helps.
>
>         The intent in the spec is to highlight that VHE is applicable to
S-EL0
>         SPs only.
>
>         With VHE, the S-EL1 exception level disappears. The SPMC can only
have
>         awareness
>         of S-EL0 SPs.
>
>         Without WHE, the S-EL1 exception level is present. As you state
above,
>         there are
>         knobs in the architecture that reduce the role of S-EL1. But
based upon
>         my
>         current understanding, the SPMC will see S-EL1 as the next lower
>         exception
>         level.
>
>     If we want OPTEE to be running a number of Apps in one Secure world
VM, we
>     wouldn't need WHE right?
>
>     Can you confirm that without enabling VHE, an OPTEE VM, and its
containing
>     apps, is possible and can communicate with  normal world VMs?
>
>
>
>         A S-EL0 SP can be "wrapped" as a S-EL1 SP such that the role of
S-EL1
>         is
>         reduced. This could be done in SW (e.g. a shim layer in S-EL1),
HW
>         (e.g. the TGE
>         bit above) or both. But architecturally, the SPMC will still see
S-EL1.
>
>         I hope this clarifies. That said, I do need to double check the
>         original intent
>         and impact of TGE. So please do correct me if I have
misinterpreted
>         anything.
>
>         cheers,
>         Achin
>
>         >
>         >
>         > Thanks
>         > Raghu
>         >
>         >
>         > -----Original Message-----
>         > From: Olivier Deprez <Olivier.Deprez@arm.com>
>         > Sent: Tuesday, December 15, 2020 12:18 AM
>         > To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com
;
>         Olivier
>         > Deprez <Olivier.Deprez@arm.com>
>         > Subject: Re: [Hafnium] VHE support
>         >
>         > Hi Raghu,
>         >
>         > one more
>         >
>         > 5/ maybe answer to 2/3/4 is that it requires an EL1-shim
embedded
>         into
>         > Hafnium which itself ERETs to a S-EL0 partition?
>         >
>         > BTW notice my questions are obviously oriented towards the
secure
>         > implementation.
>         >
>         > Regards,
>         > Olivier.
>         >
>         > ________________________________________
>         > From: Hafnium <hafnium-bounces@lists.trustedfirmware.org> on
behalf
>         of
>         > Olivier Deprez via Hafnium <hafnium@lists.trustedfirmware.org>
>         > Sent: 15 December 2020 09:10
>         > To: hafnium@lists.trustedfirmware.org; raghu.ncstate@icloud.com
>         > Subject: Re: [Hafnium] VHE support
>         >
>         > Hi Raghu,
>         >
>         > Thanks for sharing this work.
>         >
>         > Few thoughts...
>         >
>         > 1/ I guess it requires an additional change in project/
reference to
>         > eventually enable the feature for a platform (enable_vhe=1 in
the gn
>         build
>         > flow)?
>         >
>         > Sorry if my questions below sound obvious, I may miss bits of
the VHE
>         > architecture.
>         >
>         > 2/ The changes are effectively toggling HCR_EL2.E2H=1.
Currently when
>         > Hafnium ERETs resuming a SP, this happens with the secure EL1&0
>         translation
>         > regime.
>         > Though what's the next step forward?
>         > Does this require a host OS in secure world? (like linux does
when
>         booting
>         > under a Hypervisor with VHE enabled?) Would this be a
VHE-enabled
>         TOS?
>         >
>         > 3/ Is the later goal to enable the secure EL2&0 translation
regime?
>         >
>         > 4/ Is there anything to do with HCR_EL2.TGE?
>         >
>         > Regards,
>         > Olivier.
>         >
>         >
>         > ________________________________________
>         > From: Hafnium <hafnium-bounces@lists.trustedfirmware.org> on
behalf
>         of Raghu
>         > Krishnamurthy via Hafnium <hafnium@lists.trustedfirmware.org>
>         > Sent: 15 December 2020 04:57
>         > To: hafnium@lists.trustedfirmware.org
>         > Subject: [Hafnium] VHE support
>         >
>         > Hi All,
>         >
>         >
>         >
>         > I have a series of patches pushed to Gerrit at
>         > https://review.trustedfirmware.org/c/hafnium/hafnium/+/7599
with
>         topic
>         > "vhe_enable". The goal of this patch series is to enable VM's
in both
>         secure
>         > and normal world to run with VHE enabled(hcr_el2.e2h=1),
without
>         breaking
>         > any existing functionality. This is expected to be the first
step in
>         the
>         > long term goal of enabling S-EL0 partitions(and optionally EL0
>         partitions),
>         > that require VHE support, per the FF-A 1.0 Spec. I'd appreciate
>         feedback on
>         > the patches and approach taken to nominally enabling VHE. Note
that
>         the FF-A
>         > 1.0 spec(AFAIK) does not expect VHE support in the normal world
but
>         this
>         > patch series enables it anyway due to the wealth of available
tests
>         in the
>         > hafnium test suite to help with providing confidence in the
>         implementation.
>         >
>         >
>         >
>         > The patch series has been tested as follows:
>         >
>         >
>         >
>         > Hafnium tests using QEMU(prebuilt in the hafnium repo) -
Without VHE,
>         since
>         > the prebuilt QEMU does not support VHE.
>         >
>         > Hafnium tests using QEMU(5.2-RC4, built from source) - With and
>         without VHE,
>         > this version of QEMU supports VHE.
>         >
>         > Hafnium tests using FVP 11.12.28 - With and Without VHE.
>         >
>         > TFTF tests for secure hafnium using FVP 11.12.28 - With and
without
>         VHE.
>         >
>         >
>         >
>         >
>         >
>         > Thanks
>         >
>         > Raghu
>         >
>         > --
>         > Hafnium mailing list
>         > Hafnium@lists.trustedfirmware.org
>         > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>         > --
>         > Hafnium mailing list
>         > Hafnium@lists.trustedfirmware.org
>         > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>         >
>         --
>         Hafnium mailing list
>         Hafnium@lists.trustedfirmware.org
>         https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
>
>
>
>     --
>
>     [uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing
Group
>
>          T: +33.67221.6485
>          francois.ozog@linaro.org | Skype: ffozog
>
>
>
>
>
>
> --
> [uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing Group
>      T: +33.67221.6485
>      francois.ozog@linaro.org | Skype: ffozog
>

-- [uc] François-Frédéric Ozog | Director Linaro Edge & Fog Computing Group T: +33.67221.6485 francois.ozog@linaro.org | Skype: ffozog