Hi,
I noticed OP-TEE accesses CNTPCT_EL1 aka the physical system counter for different reasons (small delay loops, user session time, debug/trace,...)
I wonder how this should be supported when OP-TEE runs as a Secure Partition (and more generally for other TEEs and bare metal SPs) under the SPMC. I guess this is ok as long as SPs are all statically loaded at boot time (that is all SPs use the same t0 reference time), and thus the virtual system time is similar to the physical system time.
For a VM running under Hafnium, EL1 physical counter and timer accesses are: -not trapped when accessed from the PVM. -trapped when accessed from secondary VMs. (https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/hy...)
Hafnium (or the SPMC): -does not access the EL1 physical counter (CNTPCT), nor the EL1 physical timer (CNTP) -does not setup the virtual counter offset (CNTVOFF_EL2=0 which leads to CNTVCT=CNTPCT) -emulates the EL1 timer using virtual timer registers (CNTVCT and CNTV), and Hypervisor timer (CNTHP) for the primary VM.
Considering OP-TEE currently uses the physical counter (and does not use any architected timer), options are: -do not trap CNTPCT accesses. The problem I see here in the longer term, the virtual time does no longer match physical time (e.g. consider dynamic loading of SPs). -the SPMC traps CNTPCT accesses for all SPs (and maybe emulate like non-VHE KVM does, but this incurs latency in returning the virtual counter value). -OP-TEE replaces its CNTPCT usage by CNTVCT. The latter behaves such that if virtualization is disabled accesses are made with CNTVCT=CNTPCT. I tend to think that's the approach used by linux (needs further check though).
Let me know your thoughts (@Achin, @Andrew, @JensW, others ...)
Other considerations: -With S-EL2 enabled, the EL3 physical timer cannot be accessed from lower ELs whichever SCR_EL3.ST (the behavior is slightly different when virtualization is disabled). -S-EL2 physical (CNTHPS) and virtual (CNTHVS) timers are unused. -The EL1 virtual timer interrupt can fire only under the condition that the SP is voluntarily scheduled either by a direct message request or ffa_run. In other words such interrupt is never triggered by HW, but because the SPMC injects it to an SP vCPU which explicitly enabled it, when the virtual timer expires. - I noticed CNTPCT/CNTVCT direct reads within OP-TEE and Hafnium code bases. Arm ARM quite clearly mentions that such reads must be preceded by an ISB to forbid re-ordering. I think it's worth doing the change.
Regards, Olivier.
Hi Olivier,
On Mon, Nov 30, 2020 at 3:31 PM Olivier Deprez Olivier.Deprez@arm.com wrote:
Hi,
I noticed OP-TEE accesses CNTPCT_EL1 aka the physical system counter for different reasons (small delay loops, user session time, debug/trace,...)
I wonder how this should be supported when OP-TEE runs as a Secure Partition (and more generally for other TEEs and bare metal SPs) under the SPMC. I guess this is ok as long as SPs are all statically loaded at boot time (that is all SPs use the same t0 reference time), and thus the virtual system time is similar to the physical system time.
OP-TEE is if I'm remembering correctly only using the timer to compare with an earlier read from the timer. So initial value shouldn't matter.
For a VM running under Hafnium, EL1 physical counter and timer accesses are: -not trapped when accessed from the PVM. -trapped when accessed from secondary VMs. (https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/hy...)
Hafnium (or the SPMC): -does not access the EL1 physical counter (CNTPCT), nor the EL1 physical timer (CNTP) -does not setup the virtual counter offset (CNTVOFF_EL2=0 which leads to CNTVCT=CNTPCT) -emulates the EL1 timer using virtual timer registers (CNTVCT and CNTV), and Hypervisor timer (CNTHP) for the primary VM.
Considering OP-TEE currently uses the physical counter (and does not use any architected timer), options are: -do not trap CNTPCT accesses. The problem I see here in the longer term, the virtual time does no longer match physical time (e.g. consider dynamic loading of SPs). -the SPMC traps CNTPCT accesses for all SPs (and maybe emulate like non-VHE KVM does, but this incurs latency in returning the virtual counter value). -OP-TEE replaces its CNTPCT usage by CNTVCT. The latter behaves such that if virtualization is disabled accesses are made with CNTVCT=CNTPCT. I tend to think that's the approach used by linux (needs further check though).
I think it's fine to switch to CNTVCT where/when needed in OP-TEE.
Let me know your thoughts (@Achin, @Andrew, @JensW, others ...)
Other considerations: -With S-EL2 enabled, the EL3 physical timer cannot be accessed from lower ELs whichever SCR_EL3.ST (the behavior is slightly different when virtualization is disabled). -S-EL2 physical (CNTHPS) and virtual (CNTHVS) timers are unused. -The EL1 virtual timer interrupt can fire only under the condition that the SP is voluntarily scheduled either by a direct message request or ffa_run. In other words such interrupt is never triggered by HW, but because the SPMC injects it to an SP vCPU which explicitly enabled it, when the virtual timer expires.
- I noticed CNTPCT/CNTVCT direct reads within OP-TEE and Hafnium code bases. Arm ARM quite clearly mentions that such reads must be preceded by an ISB to forbid re-ordering. I think it's worth doing the change.
Thanks for the heads up, I'll check and update in OP-TEE.
Cheers, Jens
Regards, Olivier.
hafnium@lists.trustedfirmware.org
-
Jens Wiklander -
Olivier Deprez