I have been reviewing how OP‑TEE is implemented and how secure‑world invocations behave. The goal was to determine whether an OP‑TEE call can delay the Linux side and introduce latency depending on the time spent in the secure world.
Similar latency effects are already known for EFI runtime services, but this was not documented. To mitigate the impact, EFI runtime invocations can be restricted to specific CPUs so that real‑time workloads on other CPUs remain unaffected. This mechanism, however, is only described in the commit that introduced it.
This change adds a firmware section that documents these behaviours explicitly. It highlights cases where firmware can delay the kernel, information that may be unfamiliar to some users and surprising-or concerning-to others.
Assisted-by: Microsoft-Copilot Signed-off-by: Sebastian Andrzej Siewior bigeasy@linutronix.de ---
v1…v2: https://lore.kernel.org/all/20260701091226.7SWW4TrT@linutronix.de - Rewrote the OP-TEE bits after some feedback from Ilias and Jens. Added a link to the TF-A/OP-TEE documentation. The main difference is that in contrast to my initial belief, OP-TEE can disable normal-world's interrupts and it is not guaranteed that normal world can always preempt the secure world.
Documentation/core-api/real-time/hardware.rst | 104 ++++++++++++++++++ 1 file changed, 104 insertions(+)
diff --git a/Documentation/core-api/real-time/hardware.rst b/Documentation/core-api/real-time/hardware.rst index 19f9bb3786e03..9f95e75e6aa18 100644 --- a/Documentation/core-api/real-time/hardware.rst +++ b/Documentation/core-api/real-time/hardware.rst @@ -130,3 +130,107 @@ https://github.com/Linutronix/RTC-Testbench. The goal of this project is to validate real-time network communication. It can be thought of as a "cyclictest" for networking and also serves as a starting point for application development. + +Firmware +-------- + +The firmware often plays a significant role in system operation because it can +perform tasks that the kernel cannot directly access, and in some cases it can +even preempt or intercept the kernel. + +A common example of firmware assisting the kernel is when it provides a generic +interface to a resource. Instead of accessing an RTC chip through an I2C host +controller, the kernel may query the firmware for the current time, and the +firmware then accesses the RTC behind the scenes. + +Firmware can also intercept kernel execution by providing services that +temporarily take control of the system. One example is memory scrubbing, where +the firmware periodically pauses the kernel, reads back portions of system +memory, and then returns control. During this time, the kernel is effectively +interrupted. +In contrast, some systems provide hardware-based memory scrubbing, which +operates independently of firmware or software. See +Documentation/edac/scrub.rst for details. + +If the kernel is intercepted for longer periods then these periods can be made +visible with the hardware latency detector. See +Documentation/trace/hwlat_detector.rst. + +The kernel can also be intercepted in response to specific events, such as +overheating. In this case, the firmware may throttle the CPU or shut it down +immediately to prevent hardware damage. + +Unless the firmware is well documented, it should be thoroughly tested to +uncover any unexpected behaviour. + +EFI +~~~~ + +EFI provides runtime services that act as a communication interface between the +firmware and the operating system. One such service is reading and writing EFI +variables, which are used, for example, to determine the boot source. + +Invoking a runtime service may require the architecture to disable kernel +preemption or interrupts during the call. This means the duration of a service +invocation directly affects the system’s observable latency. There is also +nothing that prevents a service call from disabling interrupts internally while +it runs. + +For these reasons, EFI runtime services are disabled by default on a PREEMPT_RT +kernel. They can still be enabled at boot time or via a Kconfig option if +required. +The native EFI runtime service implementation (where both the EFI service and +the kernel are either 32-bit or 64-bit executables) uses a wrapper mechanism +that invokes the service through a dedicated workqueue. This workqueue is named +efi_runtime, and it can be restricted to a housekeeping CPU using the +``/sys/devices/virtual/workqueue/efi_runtime/cpumask`` sysfs file. Assigning it +to a housekeeping CPU ensures that potentially long service invocations do not +impact the real-time workload which is restricted to other CPUs. + +It must also be verified that the runtime services behave as expected. Some +implementations on the x86 architecture pause all other CPUs while one CPU +performs the service call. In such cases, the interruption affects all CPUs, +and restricting the workqueue to a single CPU provides no benefit. + +OP-TEE (ARM) +~~~~~~~~~~~~ + +Execution flows from the normal world (Linux) into the secure world (OP-TEE) +through the secure monitor at EL3. The transition is initiated by the `smc` +(Secure Monitor Call) opcode or the `hvc` (Hypervisor Call) opcode together +with a function identifier. The calling convention defines two types of calls: +**yielding calls** and **fast calls**: + +- A **yielding call** unmasks interrupts before handling the requested service, + allowing normal world interrupts to occur. +- A **fast call** handles the requested service atomically, without allowing + interrupts from either the normal world or the secure world. + +In addition, the secure world (EL3 and OP-TEE) can receive interrupts routed to +the secure world. While a secure world interrupt is being serviced, +normal world interrupts are masked and cannot preempt the operation. + +The transition from normal world to secure monitor to OP-TEE and back introduces +additional latency due to world switching and context save/restore. This +overhead is typically a few microseconds and usually remains within the noise +floor. + +It is worth noting that the normal world cannot mask secure interrupts, while +the secure world can mask normal-world interrupts during execution. How OP-TEE +affects real-time workloads depends on whether secure interrupts are enabled +and which OP-TEE services are invoked. + +A practical concern is any fast call that runs longer than expected, for +example a function that occasionally performs a long-running cryptographic +computation. Another example that may block in an unexpected way are OP-TEE +drivers that issue RPC requests. An OP-TEE service in the secure world (RPMB +for instance) may need to issue a request back to the normal world (the Linux +driver) in order to complete the operation. While Linux remains preemptible, +the thread that issued the request stays blocked until the RPC completes and +the secure function call returns. + +The TF-A project provides documentation on interrupt management: +https://trustedfirmware-a.readthedocs.io/en/latest/design/interrupt-framewor... + +The OP-TEE project provides documentation on how interrupts are handled: +https://optee.readthedocs.io/en/latest/architecture/core.html#interrupt-hand...
A gentle ping.
On 2026-07-18 19:50:41 [+0200], To linux-rt-devel@lists.linux.dev wrote:
I have been reviewing how OP‑TEE is implemented and how secure‑world invocations behave. The goal was to determine whether an OP‑TEE call can delay the Linux side and introduce latency depending on the time spent in the secure world.
Similar latency effects are already known for EFI runtime services, but this was not documented. To mitigate the impact, EFI runtime invocations can be restricted to specific CPUs so that real‑time workloads on other CPUs remain unaffected. This mechanism, however, is only described in the commit that introduced it.
This change adds a firmware section that documents these behaviours explicitly. It highlights cases where firmware can delay the kernel, information that may be unfamiliar to some users and surprising-or concerning-to others.
Assisted-by: Microsoft-Copilot Signed-off-by: Sebastian Andrzej Siewior bigeasy@linutronix.de
v1…v2: https://lore.kernel.org/all/20260701091226.7SWW4TrT@linutronix.de
- Rewrote the OP-TEE bits after some feedback from Ilias and Jens. Added a link to the TF-A/OP-TEE documentation. The main difference is that in contrast to my initial belief, OP-TEE can disable normal-world's interrupts and it is not guaranteed that normal world can always preempt the secure world.
Documentation/core-api/real-time/hardware.rst | 104 ++++++++++++++++++ 1 file changed, 104 insertions(+)
diff --git a/Documentation/core-api/real-time/hardware.rst b/Documentation/core-api/real-time/hardware.rst index 19f9bb3786e03..9f95e75e6aa18 100644 --- a/Documentation/core-api/real-time/hardware.rst +++ b/Documentation/core-api/real-time/hardware.rst @@ -130,3 +130,107 @@ https://github.com/Linutronix/RTC-Testbench. The goal of this project is to validate real-time network communication. It can be thought of as a "cyclictest" for networking and also serves as a starting point for application development.
+Firmware +--------
+The firmware often plays a significant role in system operation because it can +perform tasks that the kernel cannot directly access, and in some cases it can +even preempt or intercept the kernel.
+A common example of firmware assisting the kernel is when it provides a generic +interface to a resource. Instead of accessing an RTC chip through an I2C host +controller, the kernel may query the firmware for the current time, and the +firmware then accesses the RTC behind the scenes.
+Firmware can also intercept kernel execution by providing services that +temporarily take control of the system. One example is memory scrubbing, where +the firmware periodically pauses the kernel, reads back portions of system +memory, and then returns control. During this time, the kernel is effectively +interrupted. +In contrast, some systems provide hardware-based memory scrubbing, which +operates independently of firmware or software. See +Documentation/edac/scrub.rst for details.
+If the kernel is intercepted for longer periods then these periods can be made +visible with the hardware latency detector. See +Documentation/trace/hwlat_detector.rst.
+The kernel can also be intercepted in response to specific events, such as +overheating. In this case, the firmware may throttle the CPU or shut it down +immediately to prevent hardware damage.
+Unless the firmware is well documented, it should be thoroughly tested to +uncover any unexpected behaviour.
+EFI +~~~~
+EFI provides runtime services that act as a communication interface between the +firmware and the operating system. One such service is reading and writing EFI +variables, which are used, for example, to determine the boot source.
+Invoking a runtime service may require the architecture to disable kernel +preemption or interrupts during the call. This means the duration of a service +invocation directly affects the system’s observable latency. There is also +nothing that prevents a service call from disabling interrupts internally while +it runs.
+For these reasons, EFI runtime services are disabled by default on a PREEMPT_RT +kernel. They can still be enabled at boot time or via a Kconfig option if +required. +The native EFI runtime service implementation (where both the EFI service and +the kernel are either 32-bit or 64-bit executables) uses a wrapper mechanism +that invokes the service through a dedicated workqueue. This workqueue is named +efi_runtime, and it can be restricted to a housekeeping CPU using the +``/sys/devices/virtual/workqueue/efi_runtime/cpumask`` sysfs file. Assigning it +to a housekeeping CPU ensures that potentially long service invocations do not +impact the real-time workload which is restricted to other CPUs.
+It must also be verified that the runtime services behave as expected. Some +implementations on the x86 architecture pause all other CPUs while one CPU +performs the service call. In such cases, the interruption affects all CPUs, +and restricting the workqueue to a single CPU provides no benefit.
+OP-TEE (ARM) +~~~~~~~~~~~~
+Execution flows from the normal world (Linux) into the secure world (OP-TEE) +through the secure monitor at EL3. The transition is initiated by the `smc` +(Secure Monitor Call) opcode or the `hvc` (Hypervisor Call) opcode together +with a function identifier. The calling convention defines two types of calls: +**yielding calls** and **fast calls**:
+- A **yielding call** unmasks interrupts before handling the requested service,
- allowing normal world interrupts to occur.
+- A **fast call** handles the requested service atomically, without allowing
- interrupts from either the normal world or the secure world.
+In addition, the secure world (EL3 and OP-TEE) can receive interrupts routed to +the secure world. While a secure world interrupt is being serviced, +normal world interrupts are masked and cannot preempt the operation.
+The transition from normal world to secure monitor to OP-TEE and back introduces +additional latency due to world switching and context save/restore. This +overhead is typically a few microseconds and usually remains within the noise +floor.
+It is worth noting that the normal world cannot mask secure interrupts, while +the secure world can mask normal-world interrupts during execution. How OP-TEE +affects real-time workloads depends on whether secure interrupts are enabled +and which OP-TEE services are invoked.
+A practical concern is any fast call that runs longer than expected, for +example a function that occasionally performs a long-running cryptographic +computation. Another example that may block in an unexpected way are OP-TEE +drivers that issue RPC requests. An OP-TEE service in the secure world (RPMB +for instance) may need to issue a request back to the normal world (the Linux +driver) in order to complete the operation. While Linux remains preemptible, +the thread that issued the request stays blocked until the RPC completes and +the secure function call returns.
+The TF-A project provides documentation on interrupt management: +https://trustedfirmware-a.readthedocs.io/en/latest/design/interrupt-framewor...
+The OP-TEE project provides documentation on how interrupts are handled:
+https://optee.readthedocs.io/en/latest/architecture/core.html#interrupt-hand...
2.53.0
Sebastian Andrzej Siewior bigeasy@linutronix.de writes:
A gentle ping.
On 2026-07-18 19:50:41 [+0200], To linux-rt-devel@lists.linux.dev wrote:
I have been reviewing how OP‑TEE is implemented and how secure‑world invocations behave. The goal was to determine whether an OP‑TEE call can delay the Linux side and introduce latency depending on the time spent in the secure world.
With RT stuff, I never know when something is deemed ready, or whether I should be the one to pick it up... I have gone ahead and applied this one, hopefully that was the right thing to do :)
Thanks,
jon
On 2026-08-13 09:28:08 [-0600], Jonathan Corbet wrote:
Sebastian Andrzej Siewior bigeasy@linutronix.de writes:
A gentle ping.
On 2026-07-18 19:50:41 [+0200], To linux-rt-devel@lists.linux.dev wrote:
I have been reviewing how OP‑TEE is implemented and how secure‑world invocations behave. The goal was to determine whether an OP‑TEE call can delay the Linux side and introduce latency depending on the time spent in the secure world.
With RT stuff, I never know when something is deemed ready, or whether I should be the one to pick it up... I have gone ahead and applied this one, hopefully that was the right thing to do :)
Thank you. RT wise there is no dedicated tree so I always rely on the upstream maintainer to apply my bits.
Here, for v1 I wasn't entirely sure if what I wrote was entirely correct but then Ilias and Jens gave me a few crumbs and I am confident that the document reflects the reality :)
Thanks,
jon
Sebastian
op-tee@lists.trustedfirmware.org
-
Jonathan Corbet -
Sebastian Andrzej Siewior