Hi,
On Wed, Aug 19, 2026 at 10:37 AM Xing Loong xing.xl.loong@gmail.com wrote:
This patch adds a Linux TEE driver that communicates with MbedTEE OS, a Trusted Execution Environment for embedded systems (https://github.com/mbedtee) via a fixed shared-memory RPC protocol over architecture-specific transports:
- ARM/ARM64: SMCCC fast/yield calls for REE-to-TEE requests; TEE-to-REE notifications via a GIC SPI edge interrupt.
- RISC-V: REE submits commands by writing to a shared-memory r2t-ring that the TEE polls; the TEE notifies the REE by writing to t2r-ring and raising a platform IMSIC MSI. No REE-to-TEE interrupt is used.
The driver implements the TEE subsystem interface (tee_driver_ops) and provides GlobalPlatform TEE Client API support, dynamic shared memory registration, and tee-supplicant support.
Signed-off-by: Xing Loong xing.xl.loong@gmail.com
I took a quick look at the mbedtee project, and it appears to be a one-person hobby project. Sounds like fun, but I wonder how meaningful it is to upstream a kernel driver if you're the only user.
Cheers, Jens
Changes in v4:
- Harden ring and RPC handling against malformed input:
- Validate both ring indices; bound async ring skip and resync the reader on corrupt framing
- Enforce RPC handler modes (COMPLETE_REE async-only, supplicant commands sync-only)
- Reference-count rpc_call objects; fix use-after-free on interrupted calls; abort and free leftover calls at teardown
- Add XA_FLAGS_LOCK_IRQ to the rpc_calls xarray
- Fix supp_recv/supp_abort_all race; malformed supp_send unblocks the worker instead of hanging it
- Reject TEE-changed parameter attributes
- Report the wire error on failed supplicant requests; fail oversized requests instead of requeueing them forever
- Wait for R2T ring space with a bounded backoff; bound COMPLETE_TEE retries and stop them at teardown
- Match the supplicant context when dispatching supp_send
- Access complete_work_pending under ring_lock
- Serialize RPC submission against shutdown and reject new calls once teardown starts
- Quiesce all RPC activity before tee_device_unregister() so parked waiters cannot deadlock teardown
- Bound non-interruptible RISC-V waits so a hung TEE cannot block ioctls forever
- Track closing state on sessions; reject concurrent close/invoke/cancel races
- Quarantine TEE-owned call objects at release instead of risking a cross-world use-after-free
Changes in v3:
- No changes (driver unchanged since v2)
Changes in v2:
- Fix supp_release incorrectly aborting unclaimed requests on close
- Fix potential tee_shm double-free on supp_recv error path
- Fix async RPC ring skip leaving orphaned payload bytes
- Fix COMPLETE_TEE retry to also handle transient -ENOMEM on RISC-V
- Fix session leak on close_session allocation failure: release kernel resources before sending RPC
Documentation/tee/index.rst | 1 + Documentation/tee/mbedtee.rst | 155 ++++++ MAINTAINERS | 9 + drivers/tee/Kconfig | 3 +- drivers/tee/Makefile | 1 + drivers/tee/mbedtee/Kconfig | 20 + drivers/tee/mbedtee/Makefile | 11 + drivers/tee/mbedtee/core.c | 260 +++++++++ drivers/tee/mbedtee/mbedtee_drv.h | 289 ++++++++++ drivers/tee/mbedtee/mbedtee_msg.h | 219 ++++++++ drivers/tee/mbedtee/rpc_callee.c | 697 +++++++++++++++++++++++++ drivers/tee/mbedtee/rpc_callee_arm.c | 91 ++++ drivers/tee/mbedtee/rpc_callee_riscv.c | 203 +++++++ drivers/tee/mbedtee/rpc_caller.c | 697 +++++++++++++++++++++++++ drivers/tee/mbedtee/rpc_caller_arm.c | 70 +++ drivers/tee/mbedtee/rpc_caller_riscv.c | 238 +++++++++ drivers/tee/mbedtee/shm_pool.c | 105 ++++ drivers/tee/mbedtee/shm_pool.h | 15 + drivers/tee/mbedtee/supp.c | 311 +++++++++++ include/uapi/linux/tee.h | 1 + 20 files changed, 3395 insertions(+), 1 deletion(-) create mode 100644 Documentation/tee/mbedtee.rst create mode 100644 drivers/tee/mbedtee/Kconfig create mode 100644 drivers/tee/mbedtee/Makefile create mode 100644 drivers/tee/mbedtee/core.c create mode 100644 drivers/tee/mbedtee/mbedtee_drv.h create mode 100644 drivers/tee/mbedtee/mbedtee_msg.h create mode 100644 drivers/tee/mbedtee/rpc_callee.c create mode 100644 drivers/tee/mbedtee/rpc_callee_arm.c create mode 100644 drivers/tee/mbedtee/rpc_callee_riscv.c create mode 100644 drivers/tee/mbedtee/rpc_caller.c create mode 100644 drivers/tee/mbedtee/rpc_caller_arm.c create mode 100644 drivers/tee/mbedtee/rpc_caller_riscv.c create mode 100644 drivers/tee/mbedtee/shm_pool.c create mode 100644 drivers/tee/mbedtee/shm_pool.h create mode 100644 drivers/tee/mbedtee/supp.c
diff --git a/Documentation/tee/index.rst b/Documentation/tee/index.rst index 10c3cec..6415855 100644 --- a/Documentation/tee/index.rst +++ b/Documentation/tee/index.rst @@ -12,3 +12,4 @@ TEE Subsystem amd-tee ts-tee qtee
- mbedtee
diff --git a/Documentation/tee/mbedtee.rst b/Documentation/tee/mbedtee.rst new file mode 100644 index 0000000..dfb57b8 --- /dev/null +++ b/Documentation/tee/mbedtee.rst @@ -0,0 +1,155 @@ +.. SPDX-License-Identifier: GPL-2.0
+============================================== +MbedTEE (Trusted Execution Environment) +==============================================
+The MbedTEE driver supports MbedTEE-based TEEs on ARM TrustZone (SMC calls, +GIC SPI notifications) and RISC-V IMSIC (shared-memory polling, MSI +notifications) platforms.
+The driver requires the REE and TEE CPUs that share the RPC shared memory to +be hardware coherent. In practice, the REE/TEE CPUs are expected to be in the +same CPU cluster, with coherent caches and shared visibility for the ring and +payload buffers.
+Communication overview +======================
+The driver communicates with the TEE using a fixed shared-memory RPC protocol +layered above the architecture-specific call mechanism.
+ARM platforms (SMC) +-------------------
+On ARM and AArch64 platforms, the driver uses ARM SMC Calling Convention +(SMCCC) to issue fast calls and yield calls to the TEE:
+- **Fast calls** (MBEDTEE_RPC_OS_VERSION, MBEDTEE_RPC_SUPPORT_YIELD,
- MBEDTEE_RPC_COMPLETE_TEE) are self-contained SMC calls that do
- not rely on shared-memory RPC structures.
+- **Yield calls** post the physical address of an ``rpc_cmd`` structure
- in shared memory to the TEE. The TEE may issue RPC requests back to the
- driver before the call completes; kernel-directed RPCs are handled
- directly, while supplicant RPCs (filesystem and RPMB) are forwarded to
- tee-supplicant.
+TEE-to-REE notifications on ARM are delivered via a GIC SPI edge interrupt +specified in the device tree (``interrupts`` property).
+RISC-V platforms +----------------
+On RISC-V platforms, SMCCC is not available. Communication is split into two +directions:
+- **REE to TEE (R2T)**: The driver submits commands by writing the physical
- address of an ``rpc_cmd`` structure to a REE-to-TEE ring buffer
- (``r2t-ring``). The TEE polls that ring for new commands. No interrupt
- notification is sent; the protocol relies on TEE-side polling.
+- **TEE to REE (T2R)**: The TEE writes an ``rpc_cmd`` into the TEE-to-REE
- ring and raises an IMSIC MSI to notify the REE driver. The MSI is
- allocated at runtime via the ``msi-parent`` DT property and its identity
- is published in ``callee_imsic_id``; ``callee_hartid`` tracks the
- target hart for migration during CPU hotplug. No SBI ecall is involved.
+RISC-V REE-to-TEE polling rationale +-----------------------------------
+The REE-to-TEE direction uses polling-only on RISC-V to avoid direct +notification interrupt writes from Linux to TEE-owned interrupt files. +This design is platform-independent and does not require ownership of +TEE-only hart interrupt files. The TEE-to-REE direction remains fully +functional via standard Linux MSI notifications and does not depend on +REE-to-TEE notification latency.
+Shared memory regions +=====================
+Two or three fixed shared memory regions are described in the device tree:
+``t2r-ring``
- Ring buffer used by the TEE to post RPC request notifications to the REE
- driver. Present on all platforms.
+``t2r-shm``
- Shared memory region carrying the actual ``rpc_cmd`` payloads for
- TEE-to-REE RPCs. Present on all platforms.
+``r2t-ring``
- Ring buffer used by the REE driver to submit commands to the TEE on
- RISC-V IMSIC platforms.
+RPC protocol +============
+The TEE and REE communicate through the ``rpc_cmd`` structure in shared memory +and ring buffers::
- struct rpc_cmd {
u32 id; /* RPC function ID */u16 size; /* payload size in bytes */u8 interrupted; /* set if wait was interrupted */u8 reserved; /* explicit alignment padding */s32 ret; /* return value */u32 pad; /* explicit alignment padding */u64 waiter_id; /* sync RPC request ID echoed on completion */u64 shm; /* physical address of payload (sync RPC) */u64 data[]; /* inline payload (async RPC) */- };
- struct rpc_ringbuf {
u32 wr; /* producer write pointer */u32 rd; /* consumer read pointer */u32 callee_ready; /* callee ready flag */u32 callee_imsic_id; /* RISC-V only: IMSIC local interrupt id */u32 callee_hartid; /* RISC-V only: target hart-id for T2R notification */u32 reserved; /* padding, must be zero */u8 mem[];- };
+For RISC-V T2R MSI, one MSI message targets one hart IMSIC file at a time. +The wire-visible state is split between ``callee_imsic_id`` and +``callee_hartid`` in the ring header; Linux may retarget the MSI across +online CPUs via ``irq_set_affinity()``, and the driver updates those fields +to match the selected hart.
+Architecture diagram::
- User space Kernel TEE side
- +--------+ +--------------+
- | Client | | Trusted App |
- +--------+ +--------------+
/\ /\|| +------------+ |||| | MbedTEE- | \/|| | supplicant | +--------------+|| +------------+ | TEE Internal |\/ /\ | API |- +-------+ || +--------------+
- | TEE | || +---------+---------------+ | MbedTEE |
- | Client| || | TEE | MbedTEE | | Trusted OS |
- | API | / | subsys | client driver | +--------------+
- +-------+-------------+-----+--------+----------+----+ |
- | Generic TEE API | | RPC (cmd/ring)| |
- | IOCTL (TEE_IOC_*) | | SMC / IMSIC | |
- +---------------------------+ +---------------+--------------+
+Device tree binding +===================
+See Documentation/devicetree/bindings/firmware/mbedtee,tee.yaml for the +complete device tree binding specification, including the RISC-V +``msi-parent`` requirement.
+References +==========
+- [1] MbedTEE project: https://github.com/mbedtee +- [2] ARM SMC Calling Convention: https://developer.arm.com/architectures/system-architectures/software-standa... +- [3] RISC-V IMSIC specification: https://github.com/riscv/riscv-aia diff --git a/MAINTAINERS b/MAINTAINERS index c8d4b91..344b82b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -15920,6 +15920,15 @@ F: Documentation/devicetree/bindings/net/dsa/maxlinear,mxl862xx.yaml F: drivers/net/dsa/mxl862xx/ F: net/dsa/tag_mxl862xx.c
+MBEDTEE TEE DRIVER +M: Xing Loong xing.xl.loong@gmail.com +L: op-tee@lists.trustedfirmware.org (moderated for non-subscribers) +S: Maintained +W: https://github.com/mbedtee +F: Documentation/devicetree/bindings/firmware/mbedtee,tee.yaml +F: Documentation/tee/mbedtee.rst +F: drivers/tee/mbedtee/
MCAN DEVICE DRIVER M: Markus Schneider-Pargmann msp@baylibre.com L: linux-can@vger.kernel.org diff --git a/drivers/tee/Kconfig b/drivers/tee/Kconfig index 98c3ad0..3dc3439 100644 --- a/drivers/tee/Kconfig +++ b/drivers/tee/Kconfig @@ -2,7 +2,7 @@ # Generic Trusted Execution Environment Configuration menuconfig TEE tristate "Trusted Execution Environment support"
depends on HAVE_ARM_SMCCC || COMPILE_TEST || CPU_SUP_AMD
depends on HAVE_ARM_SMCCC || RISCV || COMPILE_TEST || CPU_SUP_AMD select CRYPTO_LIB_SHA1 select DMA_SHARED_BUFFER select GENERIC_ALLOCATOR@@ -21,5 +21,6 @@ source "drivers/tee/optee/Kconfig" source "drivers/tee/amdtee/Kconfig" source "drivers/tee/tstee/Kconfig" source "drivers/tee/qcomtee/Kconfig" +source "drivers/tee/mbedtee/Kconfig"
endif diff --git a/drivers/tee/Makefile b/drivers/tee/Makefile index 3239b91..33117ea 100644 --- a/drivers/tee/Makefile +++ b/drivers/tee/Makefile @@ -8,3 +8,4 @@ obj-$(CONFIG_OPTEE) += optee/ obj-$(CONFIG_AMDTEE) += amdtee/ obj-$(CONFIG_ARM_TSTEE) += tstee/ obj-$(CONFIG_QCOMTEE) += qcomtee/ +obj-$(CONFIG_MBEDTEE) += mbedtee/ diff --git a/drivers/tee/mbedtee/Kconfig b/drivers/tee/mbedtee/Kconfig new file mode 100644 index 0000000..b1fa38a --- /dev/null +++ b/drivers/tee/mbedtee/Kconfig @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: GPL-2.0-only +# MbedTEE Trusted Execution Environment Configuration +config MBEDTEE
tristate "MbedTEE"depends on MMU && OFdepends on ((ARM || ARM64) && HAVE_ARM_SMCCC) || (RISCV && GENERIC_MSI_IRQ)helpThis enables the MbedTEE Trusted Execution Environment (TEE)driver.The driver supports ARM TrustZone (SMC calls, GIC SPI notifications)and RISC-V IMSIC (shared-memory polling, MSI notifications), withGlobalPlatform client API via /dev/tee*.The DT node should be under a firmware node and have compatible ="mbedtee,tee" with the RPC ring buffer addresses and,for ARM/ARM64, an interrupt property for TEE-to-REE notifications.See Documentation/tee/mbedtee.rst for protocol details.If unsure, say N.diff --git a/drivers/tee/mbedtee/Makefile b/drivers/tee/mbedtee/Makefile new file mode 100644 index 0000000..309eb6d --- /dev/null +++ b/drivers/tee/mbedtee/Makefile @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_MBEDTEE) += mbedtee.o
+mbedtee-y += core.o +mbedtee-y += supp.o +mbedtee-y += shm_pool.o +mbedtee-y += rpc_caller.o rpc_callee.o +mbedtee-$(CONFIG_ARM) += rpc_caller_arm.o rpc_callee_arm.o +mbedtee-$(CONFIG_ARM64) += rpc_caller_arm.o rpc_callee_arm.o +mbedtee-$(CONFIG_RISCV) += rpc_caller_riscv.o rpc_callee_riscv.o diff --git a/drivers/tee/mbedtee/core.c b/drivers/tee/mbedtee/core.c new file mode 100644 index 0000000..4a027a7 --- /dev/null +++ b/drivers/tee/mbedtee/core.c @@ -0,0 +1,260 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/errno.h> +#include <linux/io.h> +#include <linux/module.h> +#include <linux/of.h> +#include <linux/of_address.h> +#include <linux/platform_device.h> +#include <linux/slab.h> +#include <linux/string.h> +#include <linux/tee_drv.h> +#include <linux/types.h>
+#include "mbedtee_drv.h" +#include "shm_pool.h"
+int mbedtee_get_resource(struct device_node *node,
const char *name, struct resource *res)+{
struct device_node *rmem;int idx;int ret;idx = of_property_match_string(node, "memory-region-names", name);if (idx < 0)return idx;rmem = of_parse_phandle(node, "memory-region", idx);if (!rmem)return -ENODEV;ret = of_address_to_resource(rmem, 0, res);of_node_put(rmem);if (ret)return ret;res->name = name;return 0;+}
+static void mbedtee_get_version(struct tee_device *teedev,
struct tee_ioctl_version_data *vers)+{
struct tee_ioctl_version_data v = {.impl_id = TEE_IMPL_ID_MBEDTEE,.impl_caps = 0,.gen_caps = TEE_GEN_CAP_GP | TEE_GEN_CAP_REG_MEM |TEE_GEN_CAP_MEMREF_NULL,};*vers = v;+}
+static int mbedtee_open(struct tee_context *ctx) +{
struct mbedtee_context_data *ctxdata;ctxdata = kzalloc_obj(*ctxdata, GFP_KERNEL);if (!ctxdata)return -ENOMEM;mutex_init(&ctxdata->mutex);INIT_LIST_HEAD(&ctxdata->sess_list);ctx->data = ctxdata;ctx->cap_memref_null = true;return 0;+}
+static void mbedtee_release(struct tee_context *ctx) +{
struct mbedtee_device *mbedtee = tee_get_drvdata(ctx->teedev);struct mbedtee_context_data *d = ctx->data;struct mbedtee_session *s, *n;if (!d)return;if (d->is_supp_ctx) {mbedtee_supp_release(&mbedtee->supp, ctx);} else {list_for_each_entry_safe(s, n, &d->sess_list, list_node) {u32 session_id = s->session_id;if (!mbedtee_close_session(ctx, session_id))continue;/** A non-zero return can mean the session was already* dropped (e.g. -ESHUTDOWN during teardown) or kept on a* transient error. Re-look it up: kfree(NULL) is a no-op* in the former case, a force-removal in the latter.*/mutex_lock(&d->mutex);s = mbedtee_find_session_locked(d, session_id);if (s)list_del(&s->list_node);mutex_unlock(&d->mutex);kfree(s);}}kfree(d);ctx->data = NULL;+}
+static const struct tee_driver_ops mbedtee_ops = {
.get_version = mbedtee_get_version,.open = mbedtee_open,.release = mbedtee_release,.open_session = mbedtee_open_session,.close_session = mbedtee_close_session,.invoke_func = mbedtee_invoke_func,.cancel_req = mbedtee_cancel_req,.supp_recv = mbedtee_supp_recv,.supp_send = mbedtee_supp_send,.shm_register = mbedtee_shm_register,.shm_unregister = mbedtee_shm_unregister,+};
+static const struct tee_desc mbedtee_desc = {
.name = "mbedtee",.ops = &mbedtee_ops,.owner = THIS_MODULE,+};
+static int mbedtee_probe(struct platform_device *pdev) +{
int ret;long version;long yield;struct tee_shm_pool *pool;struct tee_device *teedev;struct mbedtee_device *mbedtee;mbedtee = devm_kzalloc(&pdev->dev, sizeof(*mbedtee), GFP_KERNEL);if (!mbedtee)return -ENOMEM;mbedtee->dev = &pdev->dev;xa_init_flags(&mbedtee->rpc_calls, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_IRQ);mutex_init(&mbedtee->rpc_calls_mutex);mbedtee_supp_init(&mbedtee->supp);/** R2T must be ready before T2R is advertised: synchronous T2R RPC* work replies with COMPLETE_TEE over the R2T path.*/ret = mbedtee_r2t_init(mbedtee);if (ret != 0)goto err_supp;ret = mbedtee_rpc_init(mbedtee);if (ret != 0)goto err_r2t;version = mbedtee_rpc_fastcall(mbedtee, MBEDTEE_RPC_OS_VERSION, 0, 0, 0);if (version < 0) {dev_err(&pdev->dev, "MBEDTEE_RPC_OS_VERSION failed: %ld\n", version);ret = version;goto err_rpc;}if (!MBEDTEE_VALID_VERSION(version)) {dev_err(&pdev->dev, "mbedtee not present (version=0x%lx)\n",version);ret = -ENODEV;goto err_rpc;}yield = mbedtee_rpc_fastcall(mbedtee, MBEDTEE_RPC_SUPPORT_YIELD, 0, 0, 0);dev_info(&pdev->dev, "version: 0x%06lx yield: %ld\n", version, yield);pool = mbedtee_shm_pool_alloc_pages();if (IS_ERR(pool)) {ret = PTR_ERR(pool);goto err_rpc;}teedev = tee_device_alloc(&mbedtee_desc, &pdev->dev, pool, mbedtee);if (IS_ERR(teedev)) {ret = PTR_ERR(teedev);goto err_pool;}ret = tee_device_register(teedev);if (ret)goto err_teedev;mbedtee->teedev = teedev;mbedtee->version = version;mbedtee->yield = !!yield;mbedtee->pool = pool;platform_set_drvdata(pdev, mbedtee);dev_dbg(&pdev->dev, "client initialized\n");return 0;+err_teedev:
tee_device_unregister(teedev);+err_pool:
tee_shm_pool_free(pool);+err_rpc:
mbedtee_rpc_uninit(mbedtee);+err_r2t:
mbedtee_r2t_uninit(mbedtee);+err_supp:
mbedtee_supp_uninit(&mbedtee->supp);mbedtee_rpc_calls_release(mbedtee);return ret;+}
+static void mbedtee_remove(struct platform_device *pdev) +{
struct mbedtee_device *mbedtee = platform_get_drvdata(pdev);/** Quiesce and wake all RPC waiters before tee_device_unregister(),* which blocks until every context is released; a waiter still parked* in a completion would otherwise deadlock teardown.*/mbedtee_rpc_stop(mbedtee);tee_device_unregister(mbedtee->teedev);/** Drain T2R before tearing down R2T: queued RPC work may still need* to send COMPLETE_TEE over R2T. Destroy the supplicant mutex only* after RPC work can no longer enter mbedtee_supp_handler().*/mbedtee_rpc_uninit(mbedtee);mbedtee_r2t_uninit(mbedtee);mbedtee_supp_uninit(&mbedtee->supp);tee_shm_pool_free(mbedtee->pool);mbedtee_rpc_calls_release(mbedtee);+}
+static const struct of_device_id mbedtee_dt_match[] = {
{ .compatible = "mbedtee,tee" },{ },+}; +MODULE_DEVICE_TABLE(of, mbedtee_dt_match);
+static struct platform_driver mbedtee_driver = {
.probe = mbedtee_probe,.remove = mbedtee_remove,.driver = {.name = "mbedtee",.of_match_table = mbedtee_dt_match,},+}; +module_platform_driver(mbedtee_driver);
+MODULE_AUTHOR("Xing Loong xing.xl.loong@gmail.com"); +MODULE_DESCRIPTION("MbedTEE Trusted Execution Environment driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/tee/mbedtee/mbedtee_drv.h b/drivers/tee/mbedtee/mbedtee_drv.h new file mode 100644 index 0000000..fe2bcfa --- /dev/null +++ b/drivers/tee/mbedtee/mbedtee_drv.h @@ -0,0 +1,289 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/*
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- */
+#ifndef MBEDTEE_DRV_H +#define MBEDTEE_DRV_H
+#include <linux/types.h> +#include <linux/errno.h> +#include <linux/atomic.h> +#include <linux/of.h> +#include <linux/ioport.h> +#include <linux/tee_drv.h> +#include <linux/tee_core.h> +#include <linux/interrupt.h> +#include <linux/platform_device.h> +#include <linux/msi.h> +#include <linux/mutex.h> +#include <linux/workqueue.h> +#include <linux/cpuhotplug.h> +#include <linux/cpumask.h> +#include <linux/refcount.h> +#include <linux/xarray.h> +#include "mbedtee_msg.h" +#include "shm_pool.h"
+/*
- GlobalPlatform TEE Client API return codes/origins
- used by this driver. Only the codes actually
- referenced in the driver are defined here.
- */
+#define TEEC_SUCCESS 0x00000000 +#define TEEC_ERROR_OUT_OF_MEMORY 0xFFFF000C +#define TEEC_ERROR_COMMUNICATION 0xFFFF000E
+#define TEEC_ORIGIN_COMMS 0x00000002
+#define MBEDTEE_VERSION_MAJOR 1 +#define MBEDTEE_VALID_VERSION(x) (((x) >> 16) == MBEDTEE_VERSION_MAJOR)
+/* Maximum payload size for inline (async) RPC data. */ +#define MBEDTEE_ASYNC_RPC_DATA_MAX 256
+struct rpc_transport_ops; +struct mbedtee_rpc_call; +struct mbedtee_supp_req; +struct mbedtee_device;
+struct rpc_work {
struct work_struct work;struct mbedtee_device *mbedtee;u64 waiter_id;size_t size;bool complete_only;/* Points to ctx->rpc_data (async) or into t2r_shm (sync) */void *data;void (*func)(struct mbedtee_device *mbedtee, void *data, size_t size);+};
+struct mbedtee_t2r_ctx {
spinlock_t ring_lock; /* protects T2R ring producer/consumer state */void *t2r_shm;phys_addr_t t2r_shm_phys;resource_size_t t2r_shm_sz;struct rpc_ringbuf *t2r_ring;u32 t2r_ring_sz;u32 t2r_ring_rd;void (*rpc_handlers[MBEDTEE_RPC_MAX])(struct mbedtee_device *mbedtee,void *data, size_t size);/** Inline buffer for async RPC payloads (waiter_id == 0).* Sync RPC payloads (waiter_id != 0) live in shared memory (t2r_shm).*/u8 rpc_data[MBEDTEE_ASYNC_RPC_DATA_MAX];/** CPU-hotplug tracking for T2R interrupt delivery.* callee_cpus: shadow of cpu_online_mask maintained by the core* callee layer (rpc_callee.c) for both ARM and RISC-V; used by* ARM to feed irq_set_affinity() and by RISC-V as the gate for* MSI migration decisions.* callee_virq: single MSI virq currently pinned to one CPU* (RISC-V IMSIC only; 0 for ARM).* hp_node: per-instance node for cpuhp_setup_state_multi().*/struct cpumask callee_cpus;int callee_virq;struct hlist_node hp_node;bool cpuhp_added;bool complete_work_pending;bool pending_async;bool stopping; /* set at teardown to stop COMPLETE_TEE retries */u32 pending_size;struct workqueue_struct *rpc_wq;struct rpc_work complete_work;void (*pending_func)(struct mbedtee_device *mbedtee,void *data, size_t size);+};
+struct mbedtee_r2t_ctx {
spinlock_t lock; /* protects R2T ring write-side state */struct rpc_ringbuf *ring;u32 ring_sz;u32 ring_wr;bool stopping; /* set at teardown to abort ring-full waits */+};
+struct mbedtee_rpc_transport_ctx {
int rpc_notify_virq;struct msi_msg rpc_msi_msg;+};
+struct mbedtee_supp {
struct mutex mutex; /* serializes supp request queue and active requests */struct tee_context *ctx;bool shutting_down;struct list_head reqs;struct list_head active_reqs;struct completion reqs_c;+};
+struct mbedtee_device {
u32 version;bool yield;struct device *dev;struct tee_device *teedev;struct tee_shm_pool *pool;struct mbedtee_supp supp;struct xarray rpc_calls;struct mutex rpc_calls_mutex; /* serializes submission and shutdown */bool stopping;atomic_long_t rpc_call_seq;const struct rpc_transport_ops *rpc_ops;struct mbedtee_t2r_ctx t2r;struct mbedtee_r2t_ctx r2t;struct mbedtee_rpc_transport_ctx transport;+};
+struct mbedtee_rpc_call {
struct completion c;phys_addr_t rpc_phys;u32 state;/* One reference for the caller, one for the XArray entry. */refcount_t refs;/** Wire-format rpc_cmd with inline data[] payload.* MUST be the last field: rpc.data[] is a flexible array.*/struct rpc_cmd rpc;+};
+struct mbedtee_session {
struct list_head list_node;u32 session_id;bool closing;+};
+struct mbedtee_context_data {
bool is_supp_ctx;struct mutex mutex; /* serializes session list updates */struct list_head sess_list;+};
+struct mbedtee_session * +mbedtee_find_session_locked(struct mbedtee_context_data *ctxdata,
u32 session_id);+static inline phys_addr_t mbedtee_virt_to_phys(void *va) +{
if (is_vmalloc_or_module_addr(va))return page_to_phys(vmalloc_to_page(va)) +offset_in_page(va);return virt_to_phys(va);+}
+int mbedtee_get_resource(struct device_node *node,
const char *name, struct resource *res);+/*
- RPC transport operations -- implemented per-architecture.
- Caller (rpc_caller_{arm,riscv}.c) platform differences:
- fastcall: ARM/ARM64 - direct SMC (arm_smccc_smc), synchronous,
never returns -ENOSPC.
RISC-V - writes rpc_cmd phys addr to the r2t ring
buffer, then waits for TEE completion via the t2r
ring. Despite the "fastcall" name this is a
synchronous ring-buffer round-trip; it may return
-ENOSPC when the r2t ring is full, and the caller
must retry (see rpc_routine() for the
MBEDTEE_RPC_COMPLETE_TEE retry loop).
- yieldcall: ARM/ARM64 - one SMC posting rpc_cmd phys addr, waits
for completion.
RISC-V - writes rpc_cmd phys addr to r2t ring, waits
for completion.
- Callee (rpc_callee_{arm,riscv}.c) platform differences:
- notify: ARM/ARM64 - GIC SPI; the TEE broadcasts via GIC
hardware (GICv2 ITARGETS=0xFF, GICv3 IROUTER.IRM=1).
Ring metadata unchanged on CPU affinity events.
RISC-V - IMSIC MSI; the driver writes callee_hartid
and callee_imsic_id into the ring so the TEE knows
where to send the MSI. The affinity handler updates
both ring fields when the callee CPU goes offline.
- init: Set up the interrupt delivery mechanism (MSI on RISC-V,
GIC/OF-IRQ on ARM/ARM64) and, on RISC-V, write
ring->callee_hartid and ring->callee_imsic_id. @handler
is the hard-IRQ handler that drains the ring buffer and
dispatches RPC commands.
- uninit: Tear down and free all interrupt resources.
- update_affinity: Migrate T2R interrupt delivery to @new_cpu. Called by the
common cpuhp handler when the current callee CPU goes
offline. On RISC-V IMSIC, must update ring->callee_hartid
and ring->callee_imsic_id. ARM uses GIC hardware routing
and requires no ring update.
- The getter is implemented per-architecture in rpc_callee_{arm,riscv}.c;
- only one is linked per build.
- */
+struct rpc_transport_ops {
int (*init)(struct mbedtee_device *mbedtee, struct rpc_ringbuf *ring,irq_handler_t handler);void (*uninit)(struct mbedtee_device *mbedtee);int (*update_affinity)(struct mbedtee_device *mbedtee,unsigned int new_cpu);+};
+const struct rpc_transport_ops *mbedtee_get_rpc_transport_ops(void);
+/* rpc-callee.c */ +int mbedtee_rpc_init(struct mbedtee_device *mbedtee); +void mbedtee_rpc_stop(struct mbedtee_device *mbedtee); +void mbedtee_rpc_uninit(struct mbedtee_device *mbedtee); +irqreturn_t mbedtee_rpc_irq_handler(int irq, void *dev_id);
+/* rpc_caller.c */ +int mbedtee_rpc_call_alloc(struct mbedtee_device *mbedtee,
size_t payload_size, struct mbedtee_rpc_call **call);+void mbedtee_rpc_call_free(struct mbedtee_device *mbedtee,
struct mbedtee_rpc_call *call);+void mbedtee_rpc_complete_call(struct mbedtee_device *mbedtee,
u64 waiter_id);+void mbedtee_rpc_calls_abort(struct mbedtee_device *mbedtee); +void mbedtee_rpc_calls_release(struct mbedtee_device *mbedtee); +long mbedtee_rpc_wait_for_completion(struct mbedtee_device *mbedtee,
struct mbedtee_rpc_call *call, bool killable);+long mbedtee_rpc_wait_for_completion_timeout(struct mbedtee_device *mbedtee,
struct mbedtee_rpc_call *call,unsigned long timeout);+int mbedtee_open_session(struct tee_context *ctx,
struct tee_ioctl_open_session_arg *arg,struct tee_param *param);+int mbedtee_close_session(struct tee_context *ctx, u32 session); +int mbedtee_invoke_func(struct tee_context *ctx,
struct tee_ioctl_invoke_arg *arg,struct tee_param *param);+int mbedtee_cancel_req(struct tee_context *ctx, u32 cancel_id,
u32 session);+int mbedtee_shm_register(struct tee_context *ctx, struct tee_shm *shm,
struct page **pages, size_t num_pages, unsigned long start);+int mbedtee_shm_unregister(struct tee_context *ctx, struct tee_shm *shm);
+/* rpc-caller-{arm,riscv}.c */ +long mbedtee_rpc_yieldcall(struct mbedtee_device *mbedtee,
unsigned long fn, struct mbedtee_rpc_call *call,bool interruptible);+long mbedtee_rpc_fastcall(struct mbedtee_device *mbedtee,
unsigned long fn, unsigned long a0,unsigned long a1, unsigned long a2);+int mbedtee_r2t_init(struct mbedtee_device *mbedtee); +void mbedtee_r2t_stop(struct mbedtee_device *mbedtee); +void mbedtee_r2t_uninit(struct mbedtee_device *mbedtee);
+/* supp.c */ +void mbedtee_supp_init(struct mbedtee_supp *supp); +void mbedtee_supp_uninit(struct mbedtee_supp *supp); +void mbedtee_supp_release(struct mbedtee_supp *supp, struct tee_context *ctx); +void mbedtee_supp_abort_all(struct mbedtee_supp *supp); +void mbedtee_supp_handler(struct mbedtee_device *mbedtee,
u32 func, void *data, size_t size);+int mbedtee_supp_recv(struct tee_context *ctx, u32 *func,
u32 *num_params, struct tee_param *param);+int mbedtee_supp_send(struct tee_context *ctx, u32 ret,
u32 num_params, struct tee_param *param);+#endif /* MBEDTEE_DRV_H */ diff --git a/drivers/tee/mbedtee/mbedtee_msg.h b/drivers/tee/mbedtee/mbedtee_msg.h new file mode 100644 index 0000000..deb2d8c --- /dev/null +++ b/drivers/tee/mbedtee/mbedtee_msg.h @@ -0,0 +1,219 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/*
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- MbedTEE RPC wire protocol between REE (Linux) and TEE (MbedTEE OS).
- This header is the Linux-kernel mirror of
- mbedtee-os/drivers/rpc/include/rpc/rpc.h. Both copies MUST define
- identical struct layouts. The static_assert() checks at the bottom
- of this file (and the _Static_assert() checks in rpc.h) enforce this
- contract at compile time on every architecture.
- Type mapping (wire -> Linux kernel alias):
- uint8_t -> u8 -- boolean flag / padding byte
- uint16_t -> u16 -- byte-count field (< 64 KiB payload limit)
- uint32_t -> u32 -- 32-bit ID, counter, flag, or enum
- int32_t -> s32 -- function-defined signed result code
- uint64_t -> u64 -- physical address, kernel pointer token, or size
- Using u64 for all pointer-sized fields eliminates ABI width mismatches
- in mixed-width configurations (e.g. RV64 REE + RV32 TEE, or ARM32-LPAE
- REE + ARM32 TEE). It also prevents phys_addr_t/unsigned-long confusion
- when CONFIG_PHYS_ADDR_T_64BIT=y on a 32-bit REE kernel.
- */
+#ifndef MBEDTEE_MSG_H +#define MBEDTEE_MSG_H
+#include <linux/bits.h> +#include <linux/build_bug.h> +#include <linux/types.h>
+/*
- Fastcall / yield-call classification
- */
+#define MBEDTEE_RPC_FASTCALL BIT(31) +#define MBEDTEE_RPC_IS_FASTCALL(fn) ((fn) & MBEDTEE_RPC_FASTCALL)
+/*
- REE -> TEE fast calls
- */
+#define MBEDTEE_RPC_OS_VERSION (0x0100 | MBEDTEE_RPC_FASTCALL) +#define MBEDTEE_RPC_SUPPORT_YIELD (0x0101 | MBEDTEE_RPC_FASTCALL) +#define MBEDTEE_RPC_COMPLETE_TEE (0x0102 | MBEDTEE_RPC_FASTCALL)
+/*
- REE -> TEE yield calls
- */
+#define MBEDTEE_RPC_OPEN_SESSION 1 +#define MBEDTEE_RPC_INVOKE_SESSION 2 +#define MBEDTEE_RPC_CLOSE_SESSION 3 +#define MBEDTEE_RPC_REGISTER_SHM 5 +#define MBEDTEE_RPC_UNREGISTER_SHM 6 +#define MBEDTEE_RPC_CANCEL 7
+/*
- TEE -> REE RPC calls
- */
+#define MBEDTEE_RPC_COMPLETE_REE 0 +#define MBEDTEE_RPC_REETIME 1 +#define MBEDTEE_RPC_REEFS 2 +#define MBEDTEE_RPC_RPMB 3 +#define MBEDTEE_RPC_MAX 4
+/*
- Supplicant types (TEE -> REE -> tee-supplicant)
- */
+#define MBEDTEE_SUPP_REEFS 1 +#define MBEDTEE_SUPP_RPMB 2
+/* MbedTEE RPC protocol uses 4 KiB page units regardless of host PAGE_SIZE. */ +#define MBEDTEE_PAGE_SIZE 4096UL
+/*
- REE <-> TEE RPC call command (wire format, 32 bytes fixed header).
- Fixed layout:
- +0 id u32 - RPC function identifier
- +4 size u16 - inline payload byte count (0..65535)
- +6 interrupted u8 - set by REE when caller is interrupted
- +7 reserved u8 - must be zero (explicit alignment pad)
- +8 ret s32 - return value written by callee
- +12 pad u32 - must be zero (explicit alignment pad)
- +16 waiter_id u64 - sync RPC request ID echoed on completion
- +24 shm u64 - physical address of sync-RPC shared memory
- +32 data[] u64 - inline payload (waiter_id==0) or empty
- Yield-call contract:
- Session/control RPCs return GlobalPlatform result codes here.
- Host-local errno values must be translated before being put on the wire.
- Fast calls may use function-specific return values.
- */
+struct rpc_cmd {
u32 id;u16 size;u8 interrupted;u8 reserved;s32 ret;u32 pad;u64 waiter_id;u64 shm;u64 data[];+};
+/*
- REE -> TEE pages for GP shared memory.
- All fields are u64 so the layout is identical when REE is 64-bit
- but TEE is 32-bit (physical addresses, sizes and counts use 64 bits).
- */
+struct rpc_memref {
u64 id;u64 pages;u64 offset;u64 size;u64 cnt;+};
+/*
- REE -> TEE parameter entry: value or shared memory reference.
- */
+union rpc_tee_param {
struct rpc_memref memref;struct {u32 a;u32 b;} value;+};
+/*
- REE -> TEE parameters for RPC session operations
- (MBEDTEE_RPC_OPEN_SESSION / MBEDTEE_RPC_INVOKE_SESSION /
- MBEDTEE_RPC_CLOSE_SESSION).
- */
+struct rpc_param {
s32 session_id;u32 cmd_id;u32 ret_origin;u32 params_type;union rpc_tee_param params[4];u8 uuid[16];u8 clnt_uuid[16];+};
+/*
- REE -> TEE cancellation request (MBEDTEE_RPC_CANCEL).
- */
+struct rpc_cancel_req {
u32 session_id;u32 cancel_id;+};
+/*
- TEE <-> REE supplicant payloads for REEFS and RPMB RPCs.
- Only the fixed headers are interpreted in the Linux driver.
- */
+/*
- Common supplicant command header. Both reefs_cmd and rpmb_cmd embed this
- as their first member so the kernel can write a status code through a
- single cast to (struct supp_cmd_hdr *) without caring about the full
- payload type. The wire layout is identical on every architecture.
- */
+struct supp_cmd_hdr {
s32 ret;s32 op;+};
+struct reefs_cmd {
struct supp_cmd_hdr hdr;s32 flags;s32 fd;u64 len;u8 data[];+};
+struct rpmb_cmd {
struct supp_cmd_hdr hdr;u32 nframes;u8 data[];+};
+/*
- TEE <-> REE RPC ring buffer header (24 bytes fixed).
- */
+struct rpc_ringbuf {
u32 wr; /* producer write pointer */u32 rd; /* consumer read pointer */u32 callee_ready; /* callee ready flag */u32 callee_imsic_id; /* RISC-V only: IMSIC local interrupt id */u32 callee_hartid; /* RISC-V only: target hart-id for T2R notification */u32 reserved; /* padding, must be zero */u8 mem[];+};
+/*
- Compile-time ABI layout assertions.
- These must match the _Static_assert() checks in mbedtee-os rpc/rpc.h.
- */
+static_assert(sizeof(struct rpc_cmd) == 32,
"rpc_cmd wire size mismatch");+static_assert(offsetof(struct rpc_cmd, ret) == 8,
"rpc_cmd.ret offset mismatch");+static_assert(offsetof(struct rpc_cmd, waiter_id) == 16,
"rpc_cmd.waiter_id offset mismatch");+static_assert(offsetof(struct rpc_cmd, shm) == 24,
"rpc_cmd.shm offset mismatch");+static_assert(offsetof(struct rpc_cmd, data) == 32,
"rpc_cmd.data offset mismatch");+static_assert(sizeof(struct rpc_memref) == 40,
"rpc_memref wire size mismatch");+static_assert(sizeof(struct rpc_ringbuf) == 24,
"rpc_ringbuf header size mismatch");+static_assert(sizeof(struct rpc_param) == 208,
"rpc_param wire size mismatch");+static_assert(sizeof(struct rpc_cancel_req) == 8,
"rpc_cancel_req wire size mismatch");+#endif /* MBEDTEE_MSG_H */ diff --git a/drivers/tee/mbedtee/rpc_callee.c b/drivers/tee/mbedtee/rpc_callee.c new file mode 100644 index 0000000..258f8d1 --- /dev/null +++ b/drivers/tee/mbedtee/rpc_callee.c @@ -0,0 +1,697 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- TEE->REE callee-side: handles interrupts from TEE, processes RPC requests.
- */
+#include <linux/cpu.h> +#include <linux/cpuhotplug.h> +#include <linux/cpumask.h> +#include <linux/delay.h> +#include <linux/errno.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/jiffies.h> +#include <linux/ktime.h> +#include <linux/module.h> +#include <linux/of.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <linux/tee_drv.h> +#include <linux/workqueue.h>
+#include "mbedtee_drv.h"
+/* Give up completing a sync RPC if the TEE stops draining its ring. */ +#define MBEDTEE_RPC_COMPLETE_TIMEOUT (30 * HZ)
+/*
- CPU hotplug state shared across all mbedtee instances (registered once).
- */
+static int mbedtee_cpuhp_state = -EINVAL; +static unsigned int mbedtee_cpuhp_instances; +static DEFINE_MUTEX(mbedtee_cpuhp_mutex); +/* rpc_routine() runs on workqueue, re-drains ring on completion */ +static void rpc_routine(struct work_struct *work);
+/*
- Common CPU-online handler: add the new CPU into the eligible T2R
- delivery set and let the transport re-apply affinity.
- */
+static int mbedtee_cpu_online(unsigned int cpu, struct hlist_node *node) +{
struct mbedtee_t2r_ctx *ctx =hlist_entry(node, struct mbedtee_t2r_ctx, hp_node);struct mbedtee_device *mbedtee =container_of(ctx, struct mbedtee_device, t2r);unsigned int target_cpu;if (!ctx->t2r_ring || cpumask_test_cpu(cpu, &ctx->callee_cpus))return 0;cpumask_set_cpu(cpu, &ctx->callee_cpus);target_cpu = cpumask_first(&ctx->callee_cpus);if (mbedtee->rpc_ops && mbedtee->rpc_ops->update_affinity)mbedtee->rpc_ops->update_affinity(mbedtee, target_cpu);return 0;+}
+/*
- Common CPU-offline handler: migrates T2R interrupt delivery to another
- online CPU when the current callee CPU is being taken offline.
- The transport-specific update_affinity() callback handles the actual
- interrupt migration (e.g. irq_set_affinity + callee_hartid update for
- RISC-V IMSIC; irq_set_affinity only for ARM - no ring update needed).
- */
+static int mbedtee_cpu_offline(unsigned int cpu, struct hlist_node *node) +{
struct mbedtee_t2r_ctx *ctx =hlist_entry(node, struct mbedtee_t2r_ctx, hp_node);struct mbedtee_device *mbedtee =container_of(ctx, struct mbedtee_device, t2r);unsigned int new_cpu;if (!ctx->t2r_ring || !cpumask_test_cpu(cpu, &ctx->callee_cpus))return 0;/** Remove the dying CPU from the eligible set. If the set is now* empty (all CPUs going away at once -- highly unlikely but safe),* fall back to CPU 0 which Linux guarantees to be the last offline.*/cpumask_clear_cpu(cpu, &ctx->callee_cpus);if (cpumask_empty(&ctx->callee_cpus))cpumask_set_cpu(0, &ctx->callee_cpus);new_cpu = cpumask_first(&ctx->callee_cpus);if (mbedtee->rpc_ops && mbedtee->rpc_ops->update_affinity)mbedtee->rpc_ops->update_affinity(mbedtee, new_cpu);return 0;+}
+static inline size_t t2r_available_size(struct mbedtee_t2r_ctx *ctx) +{
u32 wr;u32 rd;u32 shm_size = ctx->t2r_ring_sz;/* Pair with producer release store after writing ring payload. */wr = smp_load_acquire(&ctx->t2r_ring->wr);rd = READ_ONCE(ctx->t2r_ring_rd);/* Corrupt indices mean the stream cannot be trusted. */if (wr > shm_size || rd > shm_size)return 0;if (wr >= rd)return wr - rd;return shm_size + wr - rd;+}
+static void t2r_ring_copy(struct mbedtee_t2r_ctx *ctx,
void *data, size_t size, u32 rd)+{
u32 remain;u32 shm_size = ctx->t2r_ring_sz;struct rpc_ringbuf *shm = ctx->t2r_ring;if (rd + size <= shm_size) {memcpy(data, &shm->mem[rd], size);} else {remain = rd + size - shm_size;memcpy(data, &shm->mem[rd], size - remain);memcpy((u8 *)data + size - remain,&shm->mem[0], remain);}+}
+static void t2r_ring_advance(struct mbedtee_t2r_ctx *ctx, size_t size) +{
u32 rd = READ_ONCE(ctx->t2r_ring_rd);u32 shm_size = ctx->t2r_ring_sz;struct rpc_ringbuf *shm = ctx->t2r_ring;if (rd + size > shm_size)rd = rd + size - shm_size;elserd += size;WRITE_ONCE(ctx->t2r_ring_rd, rd);/* Publish updated consumer index after command parsing is complete. */smp_store_release(&shm->rd, rd);+}
+/*
- Read and consume @size bytes from the TEE-to-REE ring buffer.
- Must be called with ctx->ring_lock held.
- */
+static void t2r_ring_read(struct mbedtee_t2r_ctx *ctx,
void *data, size_t size)+{
t2r_ring_copy(ctx, data, size, READ_ONCE(ctx->t2r_ring_rd));t2r_ring_advance(ctx, size);+}
+static bool rpc_queue_complete_only(struct mbedtee_t2r_ctx *ctx,
u64 waiter_id)+{
/* Called with ctx->ring_lock held. */if (ctx->complete_work_pending)return false;ctx->complete_work_pending = true;ctx->complete_work.waiter_id = waiter_id;queue_work(ctx->rpc_wq, &ctx->complete_work.work);return true;+}
+/*
- Try to pick the next RPC command from the ring buffer.
- For asynchronous RPCs the data payload follows the cmd header in the
- ring. If the payload has not fully arrived yet the cmd header is still
- consumed (it was already read) but the entry is marked incomplete so
- that the next IRQ resumes reading the payload instead of re-reading
- a header.
- For synchronous RPCs (waiter_id != 0) the work descriptor is allocated
- BEFORE the ring header is consumed. This guarantees that if the
- allocator returns NULL the ring entry is left intact so the IRQ
- handler will retry on the next wake-up rather than leaving a TEE
- thread blocked with no response.
- Returns a work descriptor on success, or NULL when no complete entry
- is available.
- */
+static struct rpc_work *rpc_pick_next(struct mbedtee_device *mbedtee,
struct mbedtee_t2r_ctx *ctx, struct rpc_work *c)+{
struct rpc_cmd cmd;void (*func)(struct mbedtee_device *mbedtee, void *data, size_t size);struct rpc_work *new_work = NULL;resource_size_t off;phys_addr_t shm_phys;u64 shm_wire;/* Resume reading payload of a previously incomplete async RPC */if (ctx->pending_async) {if (t2r_available_size(ctx) < ctx->pending_size)return NULL;ctx->pending_async = false;t2r_ring_read(ctx, c->data, ctx->pending_size);c->func = ctx->pending_func;c->size = ctx->pending_size;c->waiter_id = 0;return c;}if (t2r_available_size(ctx) < sizeof(cmd))return NULL;/** Peek at the header without advancing the ring pointer.* For sync RPCs we must pre-allocate before consuming the entry.*/memset(&cmd, 0, sizeof(cmd));t2r_ring_copy(ctx, &cmd, sizeof(cmd), READ_ONCE(ctx->t2r_ring_rd));if (cmd.id >= MBEDTEE_RPC_MAX)goto skip;func = ctx->rpc_handlers[cmd.id];if (!func)goto skip;/** The TEE sends COMPLETE_REE asynchronously and the supplicant* commands (REEFS/RPMB) synchronously. Reject inverted usage so* that blocking supplicant handlers can never run inline in IRQ* context while the ring lock is held.*/switch (cmd.id) {case MBEDTEE_RPC_COMPLETE_REE:if (cmd.waiter_id != 0)goto skip;break;case MBEDTEE_RPC_REETIME:case MBEDTEE_RPC_REEFS:case MBEDTEE_RPC_RPMB:if (cmd.waiter_id == 0)goto skip;if ((cmd.id == MBEDTEE_RPC_REEFS ||cmd.id == MBEDTEE_RPC_RPMB) &&cmd.size < sizeof(struct supp_cmd_hdr))goto skip;break;default:break;}if (cmd.waiter_id == 0) {/* Async RPC: payload follows header in the ring */if (cmd.size > sizeof(ctx->rpc_data))goto skip;/* Consume the header now that we know it is valid */t2r_ring_advance(ctx, sizeof(cmd));if (t2r_available_size(ctx) < cmd.size) {/* Mark as incomplete, resume on next iteration */ctx->pending_async = true;ctx->pending_size = cmd.size;ctx->pending_func = func;return NULL;}t2r_ring_read(ctx, c->data, cmd.size);} else {/* Sync RPC: data lives in the shared memory region */shm_wire = cmd.shm;shm_phys = (phys_addr_t)shm_wire;if ((u64)shm_phys != shm_wire)goto skip;if (shm_phys < ctx->t2r_shm_phys)goto skip;off = shm_phys - ctx->t2r_shm_phys;if (cmd.size == 0 || cmd.size > ctx->t2r_shm_sz)goto skip;if (off > ctx->t2r_shm_sz - cmd.size)goto skip;new_work = kzalloc_obj(*new_work, GFP_ATOMIC);if (!new_work) {/** Leave the header in place. The TEE re-notifies when* its sync RPC retry timer fires, so this entry is* retried on the next IRQ instead of being lost.*/return NULL;}t2r_ring_advance(ctx, sizeof(cmd));c = new_work;c->mbedtee = mbedtee;c->data = ctx->t2r_shm + off;}c->func = func;c->waiter_id = cmd.waiter_id;c->size = cmd.size;return c;+skip:
/** If this was a sync RPC (waiter_id != 0) we must notify the TEE* thread so it does not wait forever. Defer the completion fastcall* to process context because transport implementations may sleep.* If the single deferred-completion slot is already in use, leave the* header in place and retry after the pending completion has drained.*/if (cmd.waiter_id != 0 && !rpc_queue_complete_only(ctx, cmd.waiter_id))return NULL;/** Bad or unhandled entry: consume header plus any inline payload* (async RPC payloads follow the header in the ring) to keep the* ring moving. For sync RPCs payload is in t2r_shm, not the ring.** An async payload must fit the ring and be fully published;* otherwise the stream cannot be framed and no later entry can be* trusted, so drop the whole pending snapshot instead of moving* the reader past the producer.*/if (cmd.waiter_id == 0 &&(cmd.size > ctx->t2r_ring_sz - sizeof(cmd) ||t2r_available_size(ctx) < sizeof(cmd) + cmd.size)) {t2r_ring_advance(ctx, t2r_available_size(ctx));return NULL;}t2r_ring_advance(ctx, sizeof(cmd) + (cmd.waiter_id ? 0 : cmd.size));return NULL;+}
+static void rpc_drain_ring(struct mbedtee_device *mbedtee) +{
struct mbedtee_t2r_ctx *ctx = &mbedtee->t2r;struct rpc_work rw = {}, *c = NULL;unsigned long flags;if (!ctx->t2r_ring)return;spin_lock_irqsave(&ctx->ring_lock, flags);while (READ_ONCE(ctx->t2r_ring_rd) !=/* Pair with producer store-release after ring write. */smp_load_acquire(&ctx->t2r_ring->wr)) {rw.data = ctx->rpc_data;c = rpc_pick_next(mbedtee, ctx, &rw);if (!c)break;if (c->waiter_id == 0) {c->func(mbedtee, c->data, c->size);} else {INIT_WORK(&c->work, rpc_routine);queue_work(ctx->rpc_wq, &c->work);}}spin_unlock_irqrestore(&ctx->ring_lock, flags);+}
+static void rpc_routine(struct work_struct *work) +{
struct rpc_work *c = container_of(work, struct rpc_work, work);struct mbedtee_t2r_ctx *ctx = &c->mbedtee->t2r;unsigned long deadline = jiffies + MBEDTEE_RPC_COMPLETE_TIMEOUT;unsigned long flags;long ret;if (!c->complete_only)c->func(c->mbedtee, c->data, c->size);/** COMPLETE_TEE must reach the TEE thread that issued the sync RPC.* On RISC-V the notification uses a shared ring buffer that may be* transiently full when many RPCs are in flight concurrently.* Keep retrying rather than silently dropping the completion, which* would leave the TEE thread blocked in rpc_call_sync forever.* On ARM the fastcall is a direct SMC so -ENOSPC never occurs.* Stop retrying at teardown or when the deadline passes so a dead* TEE cannot hang destroy_workqueue().*/do {if (READ_ONCE(ctx->stopping))break;ret = mbedtee_rpc_fastcall(c->mbedtee, MBEDTEE_RPC_COMPLETE_TEE,(unsigned long)c->waiter_id, 0, 0);if (ret != -ENOSPC && ret != -ENOMEM)break;if (time_after_eq(jiffies, deadline)) {dev_err(c->mbedtee->dev,"COMPLETE_TEE for waiter %llu timed out\n",c->waiter_id);break;}cond_resched();} while (1);if (c != &ctx->complete_work) {kfree(c);return;}/* complete_work_pending is always accessed under ring_lock. */spin_lock_irqsave(&ctx->ring_lock, flags);ctx->complete_work_pending = false;spin_unlock_irqrestore(&ctx->ring_lock, flags);if (!READ_ONCE(ctx->stopping))rpc_drain_ring(c->mbedtee);+}
+/*
- Hard IRQ handler -- drains the TEE-to-REE ring buffer.
- Asynchronous RPCs (waiter_id == 0) are handled inline since all async
- handlers (complete(), ktime_get_real_ts64()) are non-blocking.
- Synchronous RPCs (waiter_id != 0) may block waiting for the
- tee-supplicant, so they are dispatched to a workqueue.
- */
+irqreturn_t mbedtee_rpc_irq_handler(int irq, void *dev_id) +{
struct mbedtee_device *mbedtee = dev_id;if (!mbedtee)return IRQ_NONE;rpc_drain_ring(mbedtee);return IRQ_HANDLED;+}
+static void mbedtee_rpc_complete(struct mbedtee_device *mbedtee,
void *data, size_t size)+{
u64 waiter_id;if (size < sizeof(waiter_id))return;memcpy(&waiter_id, data, sizeof(waiter_id));if (!waiter_id)return;mbedtee_rpc_complete_call(mbedtee, waiter_id);+}
+static void mbedtee_ree_time(struct mbedtee_device *mbedtee,
void *data, size_t size)+{
struct timespec64 *ts = data;if (size < sizeof(*ts))return;ktime_get_real_ts64(ts);+}
+static void mbedtee_reefs_supp(struct mbedtee_device *mbedtee,
void *data, size_t size)+{
mbedtee_supp_handler(mbedtee, MBEDTEE_SUPP_REEFS, data, size);+}
+static void mbedtee_rpmb_supp(struct mbedtee_device *mbedtee,
void *data, size_t size)+{
mbedtee_supp_handler(mbedtee, MBEDTEE_SUPP_RPMB, data, size);+}
+static void mbedtee_register_rpc(struct mbedtee_device *mbedtee,
u32 id, void (*func)(struct mbedtee_device *mbedtee,void *data, size_t size))+{
struct mbedtee_t2r_ctx *ctx = &mbedtee->t2r;if (WARN_ON(id >= MBEDTEE_RPC_MAX || !func))return;ctx->rpc_handlers[id] = func;+}
+int mbedtee_rpc_init(struct mbedtee_device *mbedtee) +{
struct mbedtee_t2r_ctx *ctx = &mbedtee->t2r;struct device_node *node = mbedtee->dev->of_node;struct resource res;bool cpuhp_state_created = false;int ret;memset(ctx, 0, sizeof(*ctx));spin_lock_init(&ctx->ring_lock);mbedtee->rpc_ops = mbedtee_get_rpc_transport_ops();if (!mbedtee->rpc_ops)return -EIO;ret = mbedtee_get_resource(node, "t2r-ring", &res);if (ret) {dev_err(mbedtee->dev, "t2r-ring not found in DT\n");return ret;}ctx->t2r_ring = memremap(res.start, resource_size(&res),MEMREMAP_WB);if (!ctx->t2r_ring) {dev_err(mbedtee->dev, "failed to map t2r ring\n");return -ENOMEM;}if (resource_size(&res) <= sizeof(struct rpc_ringbuf)) {dev_err(mbedtee->dev, "t2r-ring too small\n");memunmap(ctx->t2r_ring);ctx->t2r_ring = NULL;return -EINVAL;}ctx->t2r_ring_sz = resource_size(&res) - sizeof(struct rpc_ringbuf);/* Observe latest producer index only after ring metadata is visible. */WRITE_ONCE(ctx->t2r_ring_rd, smp_load_acquire(&ctx->t2r_ring->rd));dev_dbg(mbedtee->dev, "t2r-ring %pa\n", &res.start);ret = mbedtee_get_resource(node, "t2r-shm", &res);if (ret) {dev_err(mbedtee->dev, "t2r-shm not found in DT\n");goto err_ring;}ctx->t2r_shm_phys = res.start;ctx->t2r_shm_sz = resource_size(&res);ctx->t2r_shm = memremap(res.start, ctx->t2r_shm_sz,MEMREMAP_WB);if (!ctx->t2r_shm) {dev_err(mbedtee->dev, "failed to map t2r shm\n");ret = -ENOMEM;goto err_ring;}dev_dbg(mbedtee->dev, "t2r-shm %pa\n", &res.start);ctx->rpc_wq = alloc_workqueue("mbedtee-rpc", WQ_UNBOUND, 0);if (!ctx->rpc_wq) {dev_err(mbedtee->dev, "failed to create rpc workqueue\n");ret = -ENOMEM;goto err_shm;}ctx->complete_work.mbedtee = mbedtee;ctx->complete_work.complete_only = true;INIT_WORK(&ctx->complete_work.work, rpc_routine);mbedtee_register_rpc(mbedtee, MBEDTEE_RPC_COMPLETE_REE, mbedtee_rpc_complete);mbedtee_register_rpc(mbedtee, MBEDTEE_RPC_REETIME, mbedtee_ree_time);mbedtee_register_rpc(mbedtee, MBEDTEE_RPC_REEFS, mbedtee_reefs_supp);mbedtee_register_rpc(mbedtee, MBEDTEE_RPC_RPMB, mbedtee_rpmb_supp);/** Register cpuhp instance BEFORE enabling interrupts so that* T2R delivery migration is in place before the first IRQ fires.*/mutex_lock(&mbedtee_cpuhp_mutex);if (mbedtee_cpuhp_state < 0) {mbedtee_cpuhp_state =cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,"tee/mbedtee:rpc-callee",mbedtee_cpu_online,mbedtee_cpu_offline);if (mbedtee_cpuhp_state < 0) {ret = mbedtee_cpuhp_state;mutex_unlock(&mbedtee_cpuhp_mutex);goto err_wq;}cpuhp_state_created = true;}ret = cpuhp_state_add_instance(mbedtee_cpuhp_state, &ctx->hp_node);if (!ret) {mbedtee_cpuhp_instances++;ctx->cpuhp_added = true;} else if (cpuhp_state_created) {cpuhp_remove_multi_state(mbedtee_cpuhp_state);mbedtee_cpuhp_state = -EINVAL;}mutex_unlock(&mbedtee_cpuhp_mutex);if (ret != 0)goto err_wq;ret = mbedtee->rpc_ops->init(mbedtee, ctx->t2r_ring,mbedtee_rpc_irq_handler);if (ret != 0)goto err_cpuhp;/* Signal the TEE that the REE callee is ready to receive. */smp_store_release(&ctx->t2r_ring->callee_ready, true);return 0;+err_cpuhp:
mutex_lock(&mbedtee_cpuhp_mutex);if (mbedtee_cpuhp_state >= 0 && ctx->cpuhp_added) {cpuhp_state_remove_instance_nocalls(mbedtee_cpuhp_state,&ctx->hp_node);ctx->cpuhp_added = false;if (mbedtee_cpuhp_instances)mbedtee_cpuhp_instances--;if (!mbedtee_cpuhp_instances) {cpuhp_remove_multi_state(mbedtee_cpuhp_state);mbedtee_cpuhp_state = -EINVAL;}}mutex_unlock(&mbedtee_cpuhp_mutex);+err_wq:
destroy_workqueue(ctx->rpc_wq);ctx->rpc_wq = NULL;+err_shm:
memunmap(ctx->t2r_shm);ctx->t2r_shm = NULL;+err_ring:
memunmap(ctx->t2r_ring);ctx->t2r_ring = NULL;return ret;+}
+void mbedtee_rpc_uninit(struct mbedtee_device *mbedtee) +{
struct mbedtee_t2r_ctx *ctx;if (!mbedtee)return;ctx = &mbedtee->t2r;mbedtee_rpc_stop(mbedtee);/* Transport and workqueue resources can now be torn down safely. */mutex_lock(&mbedtee_cpuhp_mutex);if (mbedtee_cpuhp_state >= 0 && ctx->cpuhp_added) {cpuhp_state_remove_instance_nocalls(mbedtee_cpuhp_state,&ctx->hp_node);ctx->cpuhp_added = false;if (mbedtee_cpuhp_instances)mbedtee_cpuhp_instances--;if (!mbedtee_cpuhp_instances) {cpuhp_remove_multi_state(mbedtee_cpuhp_state);mbedtee_cpuhp_state = -EINVAL;}}mutex_unlock(&mbedtee_cpuhp_mutex);if (mbedtee->rpc_ops)mbedtee->rpc_ops->uninit(mbedtee);if (ctx->rpc_wq) {destroy_workqueue(ctx->rpc_wq);ctx->rpc_wq = NULL;}if (ctx->t2r_shm) {memunmap(ctx->t2r_shm);ctx->t2r_shm = NULL;}if (ctx->t2r_ring) {memunmap(ctx->t2r_ring);ctx->t2r_ring = NULL;}+}
+/*
- Quiesce all RPC activity: reject new submissions and wake every waiter.
- Idempotent -- called directly from remove() and again from rpc_uninit()
- (which the probe error path reaches without a prior stop).
- */
+void mbedtee_rpc_stop(struct mbedtee_device *mbedtee) +{
struct mbedtee_t2r_ctx *ctx;if (!mbedtee)return;ctx = &mbedtee->t2r;/* Stop advertising the REE callee before interrupt teardown. */if (ctx->t2r_ring)WRITE_ONCE(ctx->t2r_ring->callee_ready, false);WRITE_ONCE(ctx->stopping, true);mbedtee_r2t_stop(mbedtee);mbedtee_supp_abort_all(&mbedtee->supp);mbedtee_rpc_calls_abort(mbedtee);+} diff --git a/drivers/tee/mbedtee/rpc_callee_arm.c b/drivers/tee/mbedtee/rpc_callee_arm.c new file mode 100644 index 0000000..871bba5 --- /dev/null +++ b/drivers/tee/mbedtee/rpc_callee_arm.c @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- ARM/ARM64 GIC transport for the TEE->REE RPC callee.
- On ARM/ARM64 the TEE raises a rising-edge software-generated interrupt
- described by the "interrupts" property of the mbedtee DT node. This file
- obtains that virq via of_irq_get() and registers the hard IRQ handler
- supplied by the core callee layer.
- */
+#include <linux/interrupt.h> +#include <linux/cpu.h> +#include <linux/of.h> +#include <linux/of_irq.h>
+#include "mbedtee_drv.h"
+static int arm_transport_init(struct mbedtee_device *mbedtee,
struct rpc_ringbuf *ring, irq_handler_t handler)+{
struct mbedtee_t2r_ctx *ctx = &mbedtee->t2r;struct mbedtee_rpc_transport_ctx *tctx = &mbedtee->transport;int ret;tctx->rpc_notify_virq = of_irq_get(mbedtee->dev->of_node, 0);if (tctx->rpc_notify_virq <= 0)return tctx->rpc_notify_virq ? tctx->rpc_notify_virq : -ENODEV;ret = request_irq(tctx->rpc_notify_virq, handler,0, "mbedtee-rpc", mbedtee);if (ret) {dev_warn(mbedtee->dev, "request_irq %d failed: %d\n",tctx->rpc_notify_virq, ret);tctx->rpc_notify_virq = 0;return ret;}/** Spread the T2R SPI across all online CPUs in Linux.* The TEE needs no CPU information from Linux: it broadcasts* via GIC hardware (GICv2 ITARGETS=0xFF / GICv3 IROUTER.IRM=1).*/cpumask_copy(&ctx->callee_cpus, cpu_online_mask);if (irq_set_affinity(tctx->rpc_notify_virq, &ctx->callee_cpus))dev_warn(mbedtee->dev, "irq_set_affinity failed\n");dev_dbg(mbedtee->dev, "t2r-irq %d\n", tctx->rpc_notify_virq);return 0;+}
+static void arm_transport_uninit(struct mbedtee_device *mbedtee) +{
struct mbedtee_rpc_transport_ctx *tctx = &mbedtee->transport;if (tctx->rpc_notify_virq > 0) {free_irq(tctx->rpc_notify_virq, mbedtee);tctx->rpc_notify_virq = 0;}+}
+/*
- ARM T2R uses a GIC SPI: the TEE broadcasts it via GIC hardware
- (GICv2 ITARGETS=0xFF or GICv3 IROUTER.IRM=1) to any alive CPU.
- TrustZone shares physical CPUs -- no ring update is ever needed.
- @new_cpu is ignored: the IRQ is re-affined to ctx->callee_cpus which
- the core callee layer keeps in sync (dying CPU already removed before
- this callback is invoked, so the mask is always accurate).
- */
+static int arm_update_affinity(struct mbedtee_device *mbedtee,
unsigned int new_cpu)+{
struct mbedtee_rpc_transport_ctx *tctx = &mbedtee->transport;if (tctx->rpc_notify_virq > 0 &&irq_set_affinity(tctx->rpc_notify_virq, &mbedtee->t2r.callee_cpus))dev_warn(mbedtee->dev, "irq_set_affinity failed\n");return 0;+}
+static const struct rpc_transport_ops arm_transport_ops = {
.init = arm_transport_init,.uninit = arm_transport_uninit,.update_affinity = arm_update_affinity,+};
+const struct rpc_transport_ops *mbedtee_get_rpc_transport_ops(void) +{
return &arm_transport_ops;+} diff --git a/drivers/tee/mbedtee/rpc_callee_riscv.c b/drivers/tee/mbedtee/rpc_callee_riscv.c new file mode 100644 index 0000000..7ac0f8e --- /dev/null +++ b/drivers/tee/mbedtee/rpc_callee_riscv.c @@ -0,0 +1,203 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2025 Xing Loong xing.xl.loong@gmail.com
- RISC-V MSI transport for the TEE->REE RPC callee.
- On RISC-V the TEE wakes the REE by writing a Message Signalled Interrupt
- into the target hart's IMSIC (Incoming MSI Controller). This file allocates
- one platform MSI from the DT-provided MSI parent and registers the hard IRQ
- handler supplied by the core callee layer.
- CPU hotplug: the common mbedtee_cpu_offline() callback in rpc_callee.c
- detects when the callee CPU goes offline and calls riscv_update_affinity()
- to migrate the MSI to another online CPU.
- */
+#include <linux/interrupt.h> +#include <linux/msi.h> +#include <linux/of_irq.h> +#include <linux/smp.h>
+#include "mbedtee_drv.h"
+static int mbedtee_ring_set_hartid(struct mbedtee_device *mbedtee,
struct rpc_ringbuf *ring,unsigned int cpu)+{
long hartid = cpuid_to_hartid_map(cpu);if (hartid < 0) {dev_err(mbedtee->dev, "CPU%u has no valid hart-id\n", cpu);return -ERANGE;}WRITE_ONCE(ring->callee_hartid, (u32)hartid);return 0;+}
+static void mbedtee_rpc_write_msi_msg(struct msi_desc *desc_msi,
struct msi_msg *msg)+{
struct mbedtee_device *mbedtee = dev_get_drvdata(desc_msi->dev);if (mbedtee)mbedtee->transport.rpc_msi_msg = *msg;+}
+static int riscv_transport_init(struct mbedtee_device *mbedtee,
struct rpc_ringbuf *ring, irq_handler_t handler)+{
struct mbedtee_t2r_ctx *ctx = &mbedtee->t2r;const struct cpumask *eff;struct mbedtee_rpc_transport_ctx *tctx = &mbedtee->transport;unsigned int cpu;struct device *dev = mbedtee->dev;int virq;int ret;/** Set a bootstrap callee_hartid (hart 0) so the TEE can poll* even if MSI setup fails below.*/ret = mbedtee_ring_set_hartid(mbedtee, ring, 0);if (ret)return ret;dev_set_drvdata(dev, mbedtee);/** The MSI domain of an OF device is resolved only once, when the* platform device is created (of_msi_configure() in of/platform.c).* mbedtee's MSI parent is the IMSIC, whose MSI domain is created* later, when the IMSIC builtin_platform_driver probes -- typically* after this device has already been created with a NULL msi.domain.** Deferred probing alone cannot recover from this: the driver core* re-runs probe() but never re-runs of_msi_configure(), so the stale* NULL domain would persist across every retry. Re-resolve it here* and defer until the IMSIC driver is up. This mirrors the RISC-V* APLIC/IOMMU platform-MSI drivers.*/if (!dev_get_msi_domain(dev)) {of_msi_configure(dev, dev->of_node);if (!dev_get_msi_domain(dev))return -EPROBE_DEFER;}ret = platform_device_msi_init_and_alloc_irqs(dev, 1,mbedtee_rpc_write_msi_msg);if (ret) {dev_err(mbedtee->dev, "MSI alloc failed: %d\n", ret);return ret;}virq = msi_get_virq(dev, 0);if (virq <= 0) {dev_err(mbedtee->dev, "no MSI virq\n");ret = -ENOENT;goto err_msi;}ret = request_irq(virq, handler, 0, "mbedtee-rpc", mbedtee);if (ret) {dev_err(mbedtee->dev, "request_irq %d failed: %d\n", virq, ret);goto err_msi;}eff = irq_get_effective_affinity_mask(virq);cpu = eff ? cpumask_first(eff) : 0;dev_dbg(mbedtee->dev, "MSI addr 0x%x%08x data %d virq %d hart %ld\n",tctx->rpc_msi_msg.address_hi, tctx->rpc_msi_msg.address_lo,tctx->rpc_msi_msg.data, virq, cpuid_to_hartid_map(cpu));/* Inform TEE of the IMSIC identity used for T2R notifications. */ret = mbedtee_ring_set_hartid(mbedtee, ring, cpu);if (ret)goto err_irq;/* Ensure callee_hartid is visible before advertising MSI interrupt ID. */smp_store_release(&ring->callee_imsic_id, tctx->rpc_msi_msg.data);/* Track the active callee CPU for hotplug migration. */ctx->callee_virq = virq;cpumask_set_cpu(cpu, &ctx->callee_cpus);return 0;+err_irq:
free_irq(virq, mbedtee);+err_msi:
platform_device_msi_free_irqs_all(dev);return ret;+}
+static void riscv_transport_uninit(struct mbedtee_device *mbedtee) +{
struct mbedtee_t2r_ctx *ctx = &mbedtee->t2r;int virq;virq = ctx->callee_virq;ctx->callee_virq = 0;if (virq > 0)free_irq(virq, mbedtee);platform_device_msi_free_irqs_all(mbedtee->dev);+}
+/*
- Migrate the IMSIC MSI delivery to @new_cpu.
- Ordering guarantee for the TEE reader:
- Clear callee_imsic_id to 0 -- TEE skips IMSIC path while we migrate.
- Call irq_set_affinity() -- triggers mbedtee_rpc_write_msi_msg() which
updates tctx->rpc_msi_msg with the new IMSIC identity.
- Update callee_hartid (WRITE_ONCE inside mbedtee_ring_set_hartid());
no explicit barrier needed here since the release store on
callee_imsic_id in step 4 provides the ordering.
- Publish new callee_imsic_id with smp_store_release -- TEE's
smp_load_acquire on callee_imsic_id pairs with this store.- */
+static int riscv_update_affinity(struct mbedtee_device *mbedtee,
unsigned int new_cpu)+{
struct mbedtee_t2r_ctx *ctx = &mbedtee->t2r;struct mbedtee_rpc_transport_ctx *tctx = &mbedtee->transport;struct rpc_ringbuf *ring = ctx->t2r_ring;int virq = ctx->callee_virq;if (!virq)return -ENODEV;/* Step 1: prevent TEE from using the stale IMSIC identity. */smp_store_release(&ring->callee_imsic_id, 0);/* Step 2: retarget the MSI; write_msi_msg callback updates rpc_msi_msg. */if (irq_set_affinity(virq, cpumask_of(new_cpu))) {dev_warn(mbedtee->dev, "irq_set_affinity to CPU%u failed\n",new_cpu);return -EIO;}/* Step 3: update callee_hartid. */if (mbedtee_ring_set_hartid(mbedtee, ring, new_cpu))return -ERANGE;/* Step 4: publish new IMSIC id with release barrier. */smp_store_release(&ring->callee_imsic_id, tctx->rpc_msi_msg.data);dev_dbg(mbedtee->dev, "T2R callee migrated to CPU%u (hart %ld)\n",new_cpu, cpuid_to_hartid_map(new_cpu));return 0;+}
+static const struct rpc_transport_ops riscv_transport_ops = {
.init = riscv_transport_init,.uninit = riscv_transport_uninit,.update_affinity = riscv_update_affinity,+};
+const struct rpc_transport_ops *mbedtee_get_rpc_transport_ops(void) +{
return &riscv_transport_ops;+} diff --git a/drivers/tee/mbedtee/rpc_caller.c b/drivers/tee/mbedtee/rpc_caller.c new file mode 100644 index 0000000..6aeb07c --- /dev/null +++ b/drivers/tee/mbedtee/rpc_caller.c @@ -0,0 +1,697 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- REE->TEE session management and yield-call interface.
- */
+#include <linux/mm.h> +#include <linux/errno.h> +#include <linux/slab.h> +#include <linux/types.h> +#include <linux/delay.h> +#include <linux/device.h> +#include <linux/refcount.h> +#include "mbedtee_drv.h"
+#define MBEDTEE_RPC_CALL_ACTIVE 0 +#define MBEDTEE_RPC_CALL_DONE 1 +#define MBEDTEE_RPC_CALL_INTERRUPTED 2 +#define MBEDTEE_RPC_CALL_ABORTED 3
+static void mbedtee_rpc_call_put(struct mbedtee_rpc_call *call) +{
if (refcount_dec_and_test(&call->refs))kfree(call);+}
+int mbedtee_rpc_call_alloc(struct mbedtee_device *mbedtee,
size_t payload_size, struct mbedtee_rpc_call **call)+{
struct mbedtee_rpc_call *rcall;unsigned long rpc_index;int ret;if (sizeof(*rcall) + payload_size > PAGE_SIZE)return -E2BIG;rcall = kzalloc(sizeof(*rcall) + payload_size, GFP_KERNEL);if (!rcall)return -ENOMEM;init_completion(&rcall->c);rcall->state = MBEDTEE_RPC_CALL_ACTIVE;/* One reference for the caller, one for the XArray entry. */refcount_set(&rcall->refs, 2);rcall->rpc.size = payload_size;rcall->rpc.ret = -EOPNOTSUPP;rcall->rpc_phys = mbedtee_virt_to_phys(&rcall->rpc);do {rpc_index = atomic_long_inc_return(&mbedtee->rpc_call_seq);} while (rpc_index == 0);mutex_lock(&mbedtee->rpc_calls_mutex);if (mbedtee->stopping) {mutex_unlock(&mbedtee->rpc_calls_mutex);kfree(rcall);return -ESHUTDOWN;}ret = xa_insert_irq(&mbedtee->rpc_calls, rpc_index, rcall, GFP_KERNEL);mutex_unlock(&mbedtee->rpc_calls_mutex);if (ret != 0) {kfree(rcall);return ret;}rcall->rpc.waiter_id = rpc_index;*call = rcall;return 0;+}
+void mbedtee_rpc_call_free(struct mbedtee_device *mbedtee,
struct mbedtee_rpc_call *call)+{
unsigned long flags;unsigned long rpc_index;bool drop_remote = false;if (!call)return;rpc_index = (unsigned long)call->rpc.waiter_id;/** Interrupted calls remain reachable by the TEE, while aborted calls* retain a quarantined remote reference. In both cases only release the* caller reference here.*/xa_lock_irqsave(&mbedtee->rpc_calls, flags);if (call->state != MBEDTEE_RPC_CALL_INTERRUPTED &&call->state != MBEDTEE_RPC_CALL_ABORTED &&xa_load(&mbedtee->rpc_calls, rpc_index) == call) {__xa_erase(&mbedtee->rpc_calls, rpc_index);drop_remote = true;}xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);if (drop_remote)mbedtee_rpc_call_put(call);/* Drop the caller reference. */mbedtee_rpc_call_put(call);+}
+/*
- Abort all outstanding calls at teardown: erase each entry and wake its
- waiter so that no workqueue item can remain blocked in a completion
- wait when destroy_workqueue() runs.
- */
+void mbedtee_rpc_calls_abort(struct mbedtee_device *mbedtee) +{
struct mbedtee_rpc_call *call;unsigned long flags;unsigned long idx;mutex_lock(&mbedtee->rpc_calls_mutex);WRITE_ONCE(mbedtee->stopping, true);/** Hold xa_lock across the whole walk so a concurrent* mbedtee_rpc_complete_call() cannot erase entries mid-iteration.* xa_for_each() only takes RCU internally, so it does not recurse* on xa_lock.*/xa_lock_irqsave(&mbedtee->rpc_calls, flags);xa_for_each(&mbedtee->rpc_calls, idx, call) {if (xchg(&call->state, MBEDTEE_RPC_CALL_ABORTED) ==MBEDTEE_RPC_CALL_ACTIVE)complete(&call->c);}xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);mutex_unlock(&mbedtee->rpc_calls_mutex);+}
+/*
- Release the XArray at teardown. Any entries left here are calls the TEE
- never completed; it may still hold their rpc_phys, so we deliberately
- leak (quarantine) them rather than risk a cross-world use-after-free.
- */
+void mbedtee_rpc_calls_release(struct mbedtee_device *mbedtee) +{
struct mbedtee_rpc_call *call;unsigned long idx;unsigned int quarantined = 0;xa_for_each(&mbedtee->rpc_calls, idx, call) {xa_erase(&mbedtee->rpc_calls, idx);/* No quiesce ACK exists, so retain the remote reference. */quarantined++;}xa_destroy(&mbedtee->rpc_calls);mutex_destroy(&mbedtee->rpc_calls_mutex);if (quarantined)dev_warn(mbedtee->dev, "quarantined %u RPC calls at release\n",quarantined);+}
+void mbedtee_rpc_complete_call(struct mbedtee_device *mbedtee,
u64 waiter_id)+{
struct mbedtee_rpc_call *call;unsigned long rpc_index;unsigned long flags;unsigned int state;rpc_index = (unsigned long)waiter_id;if ((u64)rpc_index != waiter_id)return;xa_lock_irqsave(&mbedtee->rpc_calls, flags);call = xa_load(&mbedtee->rpc_calls, rpc_index);if (!call) {xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);return;}state = cmpxchg(&call->state, MBEDTEE_RPC_CALL_ACTIVE,MBEDTEE_RPC_CALL_DONE);__xa_erase(&mbedtee->rpc_calls, rpc_index);xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);/** Wake the waiter before dropping the XArray reference: a killed* waiter that found the entry erased may release its own reference* concurrently, so the put must be the last touch of the call.*/if (state == MBEDTEE_RPC_CALL_ACTIVE)complete(&call->c);mbedtee_rpc_call_put(call);+}
+long mbedtee_rpc_wait_for_completion(struct mbedtee_device *mbedtee,
struct mbedtee_rpc_call *call, bool killable)+{
long ret;unsigned int state;unsigned long flags;unsigned long rpc_index;if (!killable) {wait_for_completion(&call->c);return READ_ONCE(call->state) == MBEDTEE_RPC_CALL_ABORTED ?-ESHUTDOWN : 0;}rpc_index = (unsigned long)call->rpc.waiter_id;ret = wait_for_completion_killable(&call->c);if (ret == 0)return READ_ONCE(call->state) == MBEDTEE_RPC_CALL_ABORTED ?-ESHUTDOWN : 0;/** Synchronize with mbedtee_rpc_complete_call() via the* xa_lock. If complete_call already consumed this entry* before we acquired the lock, the entry is already gone* from the xarray and the TEE already completed (or freed)* the call -- treat as success since the RPC is finished.* If the entry is still present, we hold the lock so* complete_call cannot race with our state transition.*/xa_lock_irqsave(&mbedtee->rpc_calls, flags);if (xa_load(&mbedtee->rpc_calls, rpc_index) != call) {state = READ_ONCE(call->state);xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);return state == MBEDTEE_RPC_CALL_ABORTED ? -ESHUTDOWN : 0;}state = cmpxchg(&call->state, MBEDTEE_RPC_CALL_ACTIVE,MBEDTEE_RPC_CALL_INTERRUPTED);if (state == MBEDTEE_RPC_CALL_DONE) {xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);return 0;}if (state == MBEDTEE_RPC_CALL_ACTIVE) {/* Publish interruption state before the TEE waiter reads it. */smp_store_release(&call->rpc.interrupted, true);xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);return ret;}xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);return state == MBEDTEE_RPC_CALL_ABORTED ? -ESHUTDOWN : ret;+}
+/*
- Non-interruptible wait bounded by @timeout: on expiry the call is marked
- INTERRUPTED (remote ref retained) and -ETIMEDOUT is returned, so a hung
- TEE cannot block the caller forever.
- */
+long mbedtee_rpc_wait_for_completion_timeout(struct mbedtee_device *mbedtee,
struct mbedtee_rpc_call *call,unsigned long timeout)+{
unsigned long flags;unsigned long rpc_index;unsigned int state;if (wait_for_completion_timeout(&call->c, timeout))return READ_ONCE(call->state) == MBEDTEE_RPC_CALL_ABORTED ?-ESHUTDOWN : 0;rpc_index = (unsigned long)call->rpc.waiter_id;xa_lock_irqsave(&mbedtee->rpc_calls, flags);if (xa_load(&mbedtee->rpc_calls, rpc_index) != call) {state = READ_ONCE(call->state);xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);return state == MBEDTEE_RPC_CALL_ABORTED ? -ESHUTDOWN : 0;}state = cmpxchg(&call->state, MBEDTEE_RPC_CALL_ACTIVE,MBEDTEE_RPC_CALL_INTERRUPTED);if (state == MBEDTEE_RPC_CALL_ACTIVE)/* Publish interruption before the TEE observes the request. */smp_store_release(&call->rpc.interrupted, true);xa_unlock_irqrestore(&mbedtee->rpc_calls, flags);if (state == MBEDTEE_RPC_CALL_ACTIVE)return -ETIMEDOUT;return state == MBEDTEE_RPC_CALL_ABORTED ? -ESHUTDOWN : 0;+}
+/* Local helpers for encoding/decoding the 4-bit-per-parameter type field. */ +static inline u32 mbedtee_param_type_get(u32 types, unsigned int idx) +{
return (types >> (idx * 4)) & 0xF;+}
+static inline u32 mbedtee_param_type_set(u32 type, unsigned int idx) +{
return (type & 0xF) << (idx * 4);+}
+static int mbedtee_param_decode(struct tee_param *params,
size_t num_params, const struct rpc_param *rp)+{
size_t n;if (num_params > ARRAY_SIZE(rp->params))return -EINVAL;for (n = 0; n < num_params; n++) {struct tee_param *p = params + n;const union rpc_tee_param *rtp = rp->params + n;u32 attr = mbedtee_param_type_get(rp->params_type, n);/** The TEE must not change the parameter type: the generic* TEE layer releases memrefs based on the attributes the* client supplied, so a type change here would leak the* shared memory reference.*/if (attr != p->attr)return -EINVAL;switch (attr) {case TEE_IOCTL_PARAM_ATTR_TYPE_NONE:memset(&p->u, 0, sizeof(p->u));break;case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_INPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_OUTPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_INOUT:p->u.value.a = rtp->value.a;p->u.value.b = rtp->value.b;break;case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_OUTPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INOUT:p->u.memref.size = rtp->memref.size;break;default:return -EINVAL;}}return 0;+}
+static int mbedtee_param_encode(struct rpc_param *rp,
size_t num_params, const struct tee_param *params)+{
size_t n;if (num_params > ARRAY_SIZE(rp->params))return -EINVAL;rp->params_type = TEE_IOCTL_PARAM_ATTR_TYPE_NONE;for (n = 0; n < num_params; n++) {const struct tee_param *p = params + n;union rpc_tee_param *rtp = rp->params + n;rp->params_type |= mbedtee_param_type_set(p->attr, n);switch (p->attr) {case TEE_IOCTL_PARAM_ATTR_TYPE_NONE:memset(rtp, 0, sizeof(*rtp));break;case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_INPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_OUTPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_INOUT:rtp->value.a = p->u.value.a;rtp->value.b = p->u.value.b;break;case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_OUTPUT:case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INOUT:if (p->u.memref.shm)rtp->memref.id = p->u.memref.shm->sec_world_id;elsertp->memref.id = 0; /* invalid-id @ mbedtee */rtp->memref.size = p->u.memref.size;rtp->memref.offset = p->u.memref.shm_offs;break;default:return -EINVAL;}}return 0;+}
+/*
- cancel_req() and shm_{register,unregister} still return Linux errno to the
- kernel TEE core. Preserve any local transport failure in @ret; otherwise
- fold a non-success TEE-side GP status into the generic -EIO expected by
- these internal-only hooks.
- */
+static int mbedtee_rpc_gp_ret_to_errno(int ret, s32 gp_ret) +{
if (ret)return ret;return gp_ret == TEEC_SUCCESS ? 0 : -EIO;+}
+struct mbedtee_session * +mbedtee_find_session_locked(struct mbedtee_context_data *ctxdata,
u32 session_id)+{
struct mbedtee_session *sess;list_for_each_entry(sess, &ctxdata->sess_list, list_node)if (sess->session_id == session_id)return sess;return NULL;+}
+int mbedtee_open_session(struct tee_context *ctx,
struct tee_ioctl_open_session_arg *arg,struct tee_param *param)+{
int ret;u32 session_id = 0;struct mbedtee_context_data *ctxdata = ctx->data;struct mbedtee_device *mbedtee = tee_get_drvdata(ctx->teedev);struct mbedtee_rpc_call *call;struct rpc_param *rp;struct mbedtee_session *sess;sess = kzalloc_obj(*sess, GFP_KERNEL);if (!sess)return -ENOMEM;ret = mbedtee_rpc_call_alloc(mbedtee, sizeof(*rp), &call);if (ret != 0) {kfree(sess);return ret;}rp = (struct rpc_param *)call->rpc.data;ret = mbedtee_param_encode(rp, arg->num_params, param);if (ret) {kfree(sess);mbedtee_rpc_call_free(mbedtee, call);return ret;}memcpy(rp->uuid, arg->uuid, sizeof(arg->uuid));memcpy(rp->clnt_uuid, arg->clnt_uuid, sizeof(arg->clnt_uuid));rp->ret_origin = TEEC_ORIGIN_COMMS;ret = mbedtee_rpc_yieldcall(mbedtee, MBEDTEE_RPC_OPEN_SESSION, call, false);dev_dbg(mbedtee->dev, "open session ret %d gp_ret %d\n", ret,call->rpc.ret);if (ret == 0 && call->rpc.ret == TEEC_SUCCESS) {session_id = rp->session_id;sess->session_id = session_id;mutex_lock(&ctxdata->mutex);list_add(&sess->list_node, &ctxdata->sess_list);mutex_unlock(&ctxdata->mutex);} else {kfree(sess);}if (ret != 0) {arg->ret = TEEC_ERROR_COMMUNICATION;arg->ret_origin = TEEC_ORIGIN_COMMS;} else if (mbedtee_param_decode(param, arg->num_params, rp)) {arg->ret = TEEC_ERROR_COMMUNICATION;arg->ret_origin = TEEC_ORIGIN_COMMS;if (call->rpc.ret == TEEC_SUCCESS)mbedtee_close_session(ctx, session_id);} else {arg->session = rp->session_id;arg->ret = (u32)call->rpc.ret;arg->ret_origin = rp->ret_origin;}mbedtee_rpc_call_free(mbedtee, call);return 0;+}
+int mbedtee_invoke_func(struct tee_context *ctx,
struct tee_ioctl_invoke_arg *arg,struct tee_param *param)+{
int ret;struct mbedtee_context_data *ctxdata = ctx->data;struct mbedtee_device *mbedtee = tee_get_drvdata(ctx->teedev);struct mbedtee_rpc_call *call;struct rpc_param *rp;struct mbedtee_session *sess;mutex_lock(&ctxdata->mutex);sess = mbedtee_find_session_locked(ctxdata, arg->session);if (!sess || sess->closing) {mutex_unlock(&ctxdata->mutex);return -EINVAL;}mutex_unlock(&ctxdata->mutex);ret = mbedtee_rpc_call_alloc(mbedtee, sizeof(*rp), &call);if (ret != 0)return ret;rp = (struct rpc_param *)call->rpc.data;rp->session_id = arg->session;rp->cmd_id = arg->func;rp->ret_origin = TEEC_ORIGIN_COMMS;ret = mbedtee_param_encode(rp, arg->num_params, param);if (ret) {mbedtee_rpc_call_free(mbedtee, call);return ret;}ret = mbedtee_rpc_yieldcall(mbedtee, MBEDTEE_RPC_INVOKE_SESSION, call, true);if (ret != 0) {/** On -ERESTARTSYS/-ETIMEDOUT/-ESHUTDOWN the call is INTERRUPTED* or ABORTED, so free() drops only the caller reference; the* XArray keeps the remote reference until the TEE completes it.*/arg->ret = TEEC_ERROR_COMMUNICATION;arg->ret_origin = TEEC_ORIGIN_COMMS;mbedtee_rpc_call_free(mbedtee, call);return 0;}dev_dbg(mbedtee->dev, "invoke session ret %d gp_ret %d\n", ret,call->rpc.ret);if (mbedtee_param_decode(param, arg->num_params, rp)) {arg->ret = TEEC_ERROR_COMMUNICATION;arg->ret_origin = TEEC_ORIGIN_COMMS;} else {arg->ret = (u32)call->rpc.ret;arg->ret_origin = rp->ret_origin;}mbedtee_rpc_call_free(mbedtee, call);return 0;+}
+int mbedtee_close_session(struct tee_context *ctx, u32 session) +{
int ret;struct mbedtee_context_data *ctxdata = ctx->data;struct mbedtee_device *mbedtee = tee_get_drvdata(ctx->teedev);struct mbedtee_rpc_call *call;struct rpc_param *rp;struct mbedtee_session *sess;bool remove;mutex_lock(&ctxdata->mutex);sess = mbedtee_find_session_locked(ctxdata, session);if (!sess) {mutex_unlock(&ctxdata->mutex);return -EINVAL;}if (sess->closing) {mutex_unlock(&ctxdata->mutex);return -EBUSY;}sess->closing = true;mutex_unlock(&ctxdata->mutex);ret = mbedtee_rpc_call_alloc(mbedtee, sizeof(*rp), &call);if (ret != 0)goto out_session;rp = (struct rpc_param *)call->rpc.data;rp->session_id = session;ret = mbedtee_rpc_yieldcall(mbedtee, MBEDTEE_RPC_CLOSE_SESSION, call, false);mbedtee_rpc_call_free(mbedtee, call);+out_session:
remove = ret == 0 || READ_ONCE(mbedtee->stopping);mutex_lock(&ctxdata->mutex);sess = mbedtee_find_session_locked(ctxdata, session);if (sess) {if (remove)list_del(&sess->list_node);elsesess->closing = false;}mutex_unlock(&ctxdata->mutex);if (remove)kfree(sess);return ret;+}
+int mbedtee_cancel_req(struct tee_context *ctx,
u32 cancel_id, u32 session)+{
int ret;struct mbedtee_context_data *ctxdata = ctx->data;struct mbedtee_device *mbedtee = tee_get_drvdata(ctx->teedev);struct mbedtee_rpc_call *call;struct rpc_cancel_req *cancel;struct mbedtee_session *sess;mutex_lock(&ctxdata->mutex);sess = mbedtee_find_session_locked(ctxdata, session);if (!sess || sess->closing) {mutex_unlock(&ctxdata->mutex);return -EINVAL;}mutex_unlock(&ctxdata->mutex);ret = mbedtee_rpc_call_alloc(mbedtee, sizeof(*cancel), &call);if (ret != 0)return ret;cancel = (struct rpc_cancel_req *)call->rpc.data;cancel->session_id = session;cancel->cancel_id = cancel_id;ret = mbedtee_rpc_yieldcall(mbedtee, MBEDTEE_RPC_CANCEL, call, false);ret = mbedtee_rpc_gp_ret_to_errno(ret, call->rpc.ret);mbedtee_rpc_call_free(mbedtee, call);return ret;+}
+int mbedtee_shm_register(struct tee_context *ctx, struct tee_shm *shm,
struct page **pages, size_t nr_pages, unsigned long start)+{
int ret;size_t i, j;u64 *pagearray;struct mbedtee_device *mbedtee = tee_get_drvdata(ctx->teedev);struct mbedtee_rpc_call *call;struct rpc_memref *memref;/* Multiple: number of MBEDTEE_PAGE_SIZE pages per Linux page. */const size_t multiple = PAGE_SIZE / MBEDTEE_PAGE_SIZE;BUILD_BUG_ON(PAGE_SIZE < MBEDTEE_PAGE_SIZE);ret = mbedtee_rpc_call_alloc(mbedtee, sizeof(*memref), &call);if (ret != 0)return ret;memref = (struct rpc_memref *)call->rpc.data;pagearray = kcalloc(nr_pages * multiple, sizeof(*pagearray), GFP_KERNEL);if (!pagearray) {ret = -ENOMEM;goto out;}for (i = 0; i < nr_pages; i++) {for (j = 0; j < multiple; j++)pagearray[i * multiple + j] = MBEDTEE_PAGE_SIZE * j +page_to_phys(pages[i]);}memref->size = tee_shm_get_size(shm);memref->offset = tee_shm_get_page_offset(shm);memref->pages = mbedtee_virt_to_phys(pagearray);memref->cnt = nr_pages * multiple;ret = mbedtee_rpc_yieldcall(mbedtee, MBEDTEE_RPC_REGISTER_SHM, call, false);ret = mbedtee_rpc_gp_ret_to_errno(ret, call->rpc.ret);if (ret == 0)shm->sec_world_id = memref->id;kfree(pagearray);+out:
mbedtee_rpc_call_free(mbedtee, call);return ret;+}
+int mbedtee_shm_unregister(struct tee_context *ctx,
struct tee_shm *shm)+{
int ret;struct mbedtee_device *mbedtee = tee_get_drvdata(ctx->teedev);struct mbedtee_rpc_call *call;struct rpc_memref *memref;ret = mbedtee_rpc_call_alloc(mbedtee, sizeof(*memref), &call);if (ret != 0)return ret;memref = (struct rpc_memref *)call->rpc.data;memref->id = shm->sec_world_id;ret = mbedtee_rpc_yieldcall(mbedtee, MBEDTEE_RPC_UNREGISTER_SHM, call, false);ret = mbedtee_rpc_gp_ret_to_errno(ret, call->rpc.ret);mbedtee_rpc_call_free(mbedtee, call);return ret;+} diff --git a/drivers/tee/mbedtee/rpc_caller_arm.c b/drivers/tee/mbedtee/rpc_caller_arm.c new file mode 100644 index 0000000..520b57d --- /dev/null +++ b/drivers/tee/mbedtee/rpc_caller_arm.c @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- ARM RPC calls to TEE - relies on SMC
- Uses arm_smccc_smc() with SMCCC-compatible calling convention:
- Fast call : r0/x0 = fn (bit 31 set), r1-r3/x1-x3 = args.
- Yield call : r0/x0 = fn (bit 31 clear), r1/x1 = phys(rpc_cmd).
- ARM fast calls are synchronous SMC instructions.
- */
+#include <linux/mm.h> +#include <linux/errno.h> +#include <linux/slab.h> +#include <linux/types.h> +#include <linux/delay.h> +#include <linux/device.h> +#include <linux/arm-smccc.h> +#include "mbedtee_drv.h"
+long mbedtee_rpc_yieldcall(struct mbedtee_device *mbedtee,
unsigned long fn, struct mbedtee_rpc_call *call,bool interruptible)+{
long ret;struct arm_smccc_res res;if (MBEDTEE_RPC_IS_FASTCALL(fn))return -EINVAL;call->rpc.id = fn;arm_smccc_smc(fn, call->rpc_phys,0, 0, 0, 0, 0, 0, &res);ret = res.a0;if (ret != 0)return ret;return mbedtee_rpc_wait_for_completion(mbedtee, call, interruptible);+}
+long mbedtee_rpc_fastcall(struct mbedtee_device *mbedtee,
unsigned long fn, unsigned long a0,unsigned long a1, unsigned long a2)+{
struct arm_smccc_res res;if (!MBEDTEE_RPC_IS_FASTCALL(fn))return -EINVAL;arm_smccc_smc(fn, a0, a1, a2, 0, 0, 0, 0, &res);return (long)res.a0;+}
+/*
- ARM uses direct SMC; no caller-side ring buffer initialisation needed.
- */
+int mbedtee_r2t_init(struct mbedtee_device *mbedtee) +{
return 0;+}
+void mbedtee_r2t_stop(struct mbedtee_device *mbedtee) +{ +}
+void mbedtee_r2t_uninit(struct mbedtee_device *mbedtee) +{ +} diff --git a/drivers/tee/mbedtee/rpc_caller_riscv.c b/drivers/tee/mbedtee/rpc_caller_riscv.c new file mode 100644 index 0000000..98f8cd3 --- /dev/null +++ b/drivers/tee/mbedtee/rpc_caller_riscv.c @@ -0,0 +1,238 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2025 Xing Loong xing.xl.loong@gmail.com
- REE->TEE RPC calls for RISC-V (IMSIC)
- */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/mm.h> +#include <linux/errno.h> +#include <linux/slab.h> +#include <linux/types.h> +#include <linux/delay.h> +#include <linux/device.h> +#include <linux/jiffies.h> +#include <linux/of_address.h> +#include <linux/io.h> +#include <linux/sizes.h>
+#include "mbedtee_drv.h"
+/*
- Time the writer may wait for ring space before giving up. Transient
- fullness is common under high concurrency; permanent fullness means
- the TEE is no longer draining and must not hang ioctls forever.
- */
+#define MBEDTEE_R2T_FULL_TIMEOUT (10 * HZ) +#define MBEDTEE_R2T_CALL_TIMEOUT (30 * HZ)
+static bool rpc_ring_enough(struct mbedtee_r2t_ctx *ctx, size_t size) +{
u32 wr;u32 rd;u32 remain;wr = READ_ONCE(ctx->ring_wr);/* Pair with callee release store when publishing ring->rd */rd = smp_load_acquire(&ctx->ring->rd);if (rd > ctx->ring_sz || wr > ctx->ring_sz)return false;if (rd <= wr)remain = ctx->ring_sz + rd - wr;elseremain = rd - wr;return remain > size;+}
+static void rpc_ring_write(struct mbedtee_r2t_ctx *ctx,
void *data, size_t size)+{
struct rpc_ringbuf *shm = ctx->ring;u32 remain;u32 wr = READ_ONCE(ctx->ring_wr);if (wr + size > ctx->ring_sz) {remain = wr + size - ctx->ring_sz;memcpy(&shm->mem[wr], data, size - remain);memcpy(&shm->mem[0], (char *)data + size - remain, remain);wr = remain;} else {memcpy(&shm->mem[wr], data, size);wr += size;}WRITE_ONCE(ctx->ring_wr, wr);/* Publish writer index after payload bytes become visible. */smp_store_release(&shm->wr, wr);+}
+/*
- Wait for @size bytes of space in the R2T ring, then write @data
- atomically under the ring lock. Returns 0 on success.
- */
+static int r2t_ring_write_wait(struct mbedtee_r2t_ctx *ctx,
void *data, size_t size)+{
unsigned long deadline = jiffies + MBEDTEE_R2T_FULL_TIMEOUT;unsigned long flags;for (;;) {spin_lock_irqsave(&ctx->lock, flags);if (READ_ONCE(ctx->stopping)) {spin_unlock_irqrestore(&ctx->lock, flags);return -ESHUTDOWN;}if (rpc_ring_enough(ctx, size)) {rpc_ring_write(ctx, data, size);spin_unlock_irqrestore(&ctx->lock, flags);return 0;}spin_unlock_irqrestore(&ctx->lock, flags);if (time_after_eq(jiffies, deadline))return -ENOSPC;usleep_range(1000, 2000);}+}
+long mbedtee_rpc_yieldcall(struct mbedtee_device *mbedtee,
unsigned long fn, struct mbedtee_rpc_call *call,bool interruptible)+{
struct mbedtee_r2t_ctx *ctx = &mbedtee->r2t;u64 phys;int ret;if (!ctx->ring)return -ENXIO;if (MBEDTEE_RPC_IS_FASTCALL(fn))return -EINVAL;call->rpc.id = fn;phys = call->rpc_phys;ret = r2t_ring_write_wait(ctx, &phys, sizeof(phys));if (ret != 0) {dev_err_ratelimited(mbedtee->dev, "rpc ring full\n");return ret;}if (interruptible)return mbedtee_rpc_wait_for_completion(mbedtee, call, true);return mbedtee_rpc_wait_for_completion_timeout(mbedtee, call,MBEDTEE_R2T_CALL_TIMEOUT);+}
+long mbedtee_rpc_fastcall(struct mbedtee_device *mbedtee,
unsigned long fn, unsigned long a0,unsigned long a1, unsigned long a2)+{
struct mbedtee_r2t_ctx *ctx = &mbedtee->r2t;struct mbedtee_rpc_call *call;u64 phys;int ret;if (!ctx->ring)return -ENXIO;if (!MBEDTEE_RPC_IS_FASTCALL(fn))return -EINVAL;ret = mbedtee_rpc_call_alloc(mbedtee, 3 * sizeof(u64), &call);if (ret != 0)return ret;call->rpc.id = fn;call->rpc.size = 3 * sizeof(u64);call->rpc.data[0] = a0;call->rpc.data[1] = a1;call->rpc.data[2] = a2;phys = call->rpc_phys;ret = r2t_ring_write_wait(ctx, &phys, sizeof(phys));if (ret != 0) {dev_err_ratelimited(mbedtee->dev, "rpc ring full\n");mbedtee_rpc_call_free(mbedtee, call);return ret;}ret = mbedtee_rpc_wait_for_completion_timeout(mbedtee, call,MBEDTEE_R2T_CALL_TIMEOUT);if (ret != 0) {mbedtee_rpc_call_free(mbedtee, call);return ret;}ret = call->rpc.ret;mbedtee_rpc_call_free(mbedtee, call);return ret;+}
+int mbedtee_r2t_init(struct mbedtee_device *mbedtee) +{
struct mbedtee_r2t_ctx *ctx = &mbedtee->r2t;struct device_node *node = mbedtee->dev->of_node;struct resource res;int ret;memset(ctx, 0, sizeof(*ctx));spin_lock_init(&ctx->lock);ret = mbedtee_get_resource(node, "r2t-ring", &res);if (ret)return ret;if (resource_size(&res) <= sizeof(struct rpc_ringbuf)) {dev_err(mbedtee->dev, "r2t-ring too small\n");return -EINVAL;}ctx->ring = memremap(res.start, resource_size(&res),MEMREMAP_WB);if (!ctx->ring) {dev_err(mbedtee->dev, "failed to map r2t ring at %pa\n",&res.start);return -ENOMEM;}ctx->ring_sz = resource_size(&res) - sizeof(struct rpc_ringbuf);/* Read initial writer index with acquire for coherent ring bootstrap. */WRITE_ONCE(ctx->ring_wr, smp_load_acquire(&ctx->ring->wr));dev_dbg(mbedtee->dev, "r2t-ring %pa\n", &res.start);return 0;+}
+void mbedtee_r2t_stop(struct mbedtee_device *mbedtee) +{
struct mbedtee_r2t_ctx *ctx = &mbedtee->r2t;unsigned long flags;spin_lock_irqsave(&ctx->lock, flags);WRITE_ONCE(ctx->stopping, true);spin_unlock_irqrestore(&ctx->lock, flags);+}
+void mbedtee_r2t_uninit(struct mbedtee_device *mbedtee) +{
struct mbedtee_r2t_ctx *ctx;if (!mbedtee)return;ctx = &mbedtee->r2t;if (ctx->ring) {/* r2t_init() failure lands here with no prior stop; flag first. */mbedtee_r2t_stop(mbedtee);memunmap(ctx->ring);ctx->ring = NULL;ctx->ring_sz = 0;ctx->ring_wr = 0;}+} diff --git a/drivers/tee/mbedtee/shm_pool.c b/drivers/tee/mbedtee/shm_pool.c new file mode 100644 index 0000000..708ab24 --- /dev/null +++ b/drivers/tee/mbedtee/shm_pool.c @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2015, Linaro Limited
- Copyright (c) 2017, EPAM Systems
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- */
+#include <linux/device.h> +#include <linux/genalloc.h> +#include <linux/slab.h>
+#include "mbedtee_drv.h"
+static int pool_op_alloc(struct tee_shm_pool *pool,
struct tee_shm *shm, size_t size, size_t align)+{
unsigned int order = get_order(size);struct page *page;int rc = 0;page = alloc_pages(GFP_KERNEL | __GFP_ZERO, order);if (!page)return -ENOMEM;shm->kaddr = page_address(page);shm->paddr = page_to_phys(page);shm->size = PAGE_SIZE << order;if (!(shm->flags & TEE_SHM_PRIV)) {struct mbedtee_device *mbedtee = tee_get_drvdata(shm->ctx->teedev);if (mbedtee->yield) {unsigned int nr_pages = 1 << order;unsigned int i;struct page **pages;pages = kmalloc_array(nr_pages, sizeof(*pages), GFP_KERNEL);if (!pages) {rc = -ENOMEM;goto err_pages;}for (i = 0; i < nr_pages; i++)pages[i] = page + i;rc = mbedtee_shm_register(shm->ctx, shm, pages,nr_pages,(unsigned long)shm->kaddr);kfree(pages);if (rc)goto err_pages;}}return 0;+err_pages:
free_pages((unsigned long)shm->kaddr, get_order(shm->size));shm->kaddr = NULL;return rc;+}
+static void pool_op_free(struct tee_shm_pool *pool,
struct tee_shm *shm)+{
if (!(shm->flags & TEE_SHM_PRIV)) {struct mbedtee_device *mbedtee = tee_get_drvdata(shm->ctx->teedev);if (mbedtee->yield && shm->sec_world_id)mbedtee_shm_unregister(shm->ctx, shm);}free_pages((unsigned long)shm->kaddr, get_order(shm->size));shm->kaddr = NULL;+}
+static void pool_op_destroy_pool(struct tee_shm_pool *pool) +{
kfree(pool);+}
+static const struct tee_shm_pool_ops pool_ops = {
.alloc = pool_op_alloc,.free = pool_op_free,.destroy_pool = pool_op_destroy_pool,+};
+/**
- mbedtee_shm_pool_alloc_pages() - create page-based allocator pool
- This pool is used when MbedTEE supports dynamic shared memory. Command
- buffers and similar structures are allocated from kernel's own memory.
- Return: pointer to a tee_shm_pool or ERR_PTR on failure
- */
+struct tee_shm_pool *mbedtee_shm_pool_alloc_pages(void) +{
struct tee_shm_pool *pool = kzalloc_obj(*pool, GFP_KERNEL);if (!pool)return ERR_PTR(-ENOMEM);pool->ops = &pool_ops;return pool;+} diff --git a/drivers/tee/mbedtee/shm_pool.h b/drivers/tee/mbedtee/shm_pool.h new file mode 100644 index 0000000..4510676 --- /dev/null +++ b/drivers/tee/mbedtee/shm_pool.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/*
- Copyright (c) 2015, Linaro Limited
- Copyright (c) 2016, EPAM Systems
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- */
+#ifndef SHM_POOL_H +#define SHM_POOL_H
+#include <linux/tee_drv.h>
+struct tee_shm_pool *mbedtee_shm_pool_alloc_pages(void);
+#endif diff --git a/drivers/tee/mbedtee/supp.c b/drivers/tee/mbedtee/supp.c new file mode 100644 index 0000000..1408403 --- /dev/null +++ b/drivers/tee/mbedtee/supp.c @@ -0,0 +1,311 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- Copyright (c) 2015, Linaro Limited
- Copyright (c) 2020 Xing Loong xing.xl.loong@gmail.com
- */
+#include <linux/device.h> +#include <linux/sched.h> +#include <linux/slab.h> +#include <linux/uaccess.h>
+#include "mbedtee_drv.h"
+struct mbedtee_supp_req {
u32 func;struct list_head node;struct task_struct *worker;struct tee_context *ctx; /* owning supplicant context */struct tee_shm *shm;void *data;size_t size;struct completion c;+};
+static int mbedtee_supp_check_recv_params(size_t num_params,
struct tee_param *params)+{
if (num_params != 1)return -EINVAL;if (!tee_param_is_memref(params) || !params->u.memref.shm)return -EINVAL;return 0;+}
+void mbedtee_supp_init(struct mbedtee_supp *supp) +{
memset(supp, 0, sizeof(*supp));mutex_init(&supp->mutex);init_completion(&supp->reqs_c);INIT_LIST_HEAD(&supp->reqs);INIT_LIST_HEAD(&supp->active_reqs);+}
+void mbedtee_supp_uninit(struct mbedtee_supp *supp) +{
mutex_destroy(&supp->mutex);+}
+static void mbedtee_supp_complete_req(struct mbedtee_supp_req *r) +{
list_del_init(&r->node);if (r->shm)tee_shm_put(r->shm);r->shm = NULL;/* Tell the TEE that the request failed, not stale success data. */if (r->data && r->size >= sizeof(struct supp_cmd_hdr))((struct supp_cmd_hdr *)r->data)->ret = TEEC_ERROR_COMMUNICATION;complete(&r->c);+}
+void mbedtee_supp_release(struct mbedtee_supp *supp, struct tee_context *ctx) +{
struct mbedtee_supp_req *r, *n;mutex_lock(&supp->mutex);/** Cancel only requests owned by @ctx. Unclaimed requests and* requests owned by other supplicant contexts are left alone.*/list_for_each_entry_safe(r, n, &supp->reqs, node) {if (r->ctx != ctx)continue;mbedtee_supp_complete_req(r);}list_for_each_entry_safe(r, n, &supp->active_reqs, node) {if (r->ctx != ctx)continue;mbedtee_supp_complete_req(r);}if (supp->ctx == ctx)supp->ctx = NULL;mutex_unlock(&supp->mutex);+}
+void mbedtee_supp_abort_all(struct mbedtee_supp *supp) +{
struct mbedtee_supp_req *r, *n;mutex_lock(&supp->mutex);supp->shutting_down = true;list_for_each_entry_safe(r, n, &supp->reqs, node)mbedtee_supp_complete_req(r);list_for_each_entry_safe(r, n, &supp->active_reqs, node)mbedtee_supp_complete_req(r);supp->ctx = NULL;mutex_unlock(&supp->mutex);complete_all(&supp->reqs_c);+}
+static int supp_enqueue_req(struct mbedtee_supp *supp,
struct mbedtee_supp_req *req)+{
int ret = 0;mutex_lock(&supp->mutex);if (supp->shutting_down)ret = -ESHUTDOWN;elselist_add_tail(&req->node, &supp->reqs);mutex_unlock(&supp->mutex);return ret;+}
+/*
- Receive an RPC request from TEE and dispatch it to supplicant.
- Blocks until the supplicant sends back the result.
- */
+void mbedtee_supp_handler(struct mbedtee_device *mbedtee,
u32 func, void *data, size_t size)+{
struct mbedtee_supp *supp = &mbedtee->supp;struct mbedtee_supp_req *req = kzalloc_obj(*req, GFP_KERNEL);struct supp_cmd_hdr *cmd = data;int ret;if (!req) {if (cmd && size >= sizeof(*cmd))cmd->ret = TEEC_ERROR_OUT_OF_MEMORY;return;}init_completion(&req->c);req->func = func;req->data = data;req->size = size;ret = supp_enqueue_req(supp, req);if (ret != 0) {if (cmd && size >= sizeof(*cmd))cmd->ret = TEEC_ERROR_COMMUNICATION;kfree(req);return;}/* Wake up the supplicant daemon to handle this request. */complete(&supp->reqs_c);/** Wait for the supplicant to process and return the result, like* OP-TEE does. If the supplicant dies, mbedtee_supp_release()* completes the request, so this cannot block forever.*/wait_for_completion(&req->c);kfree(req);+}
+/*
- Called by supplicant to receive the next pending request.
- */
+int mbedtee_supp_recv(struct tee_context *ctx,
u32 *func, u32 *num_params, struct tee_param *param)+{
struct tee_device *teedev = ctx->teedev;struct mbedtee_device *mbedtee = tee_get_drvdata(teedev);struct mbedtee_supp *supp = &mbedtee->supp;struct mbedtee_supp_req *req;struct mbedtee_context_data *d = ctx->data;struct tee_shm *shm;int ret;ret = mbedtee_supp_check_recv_params(*num_params, param);if (ret != 0)return ret;shm = param->u.memref.shm;shm = tee_shm_get_from_id(ctx, shm->id);tee_shm_put(param->u.memref.shm);if (IS_ERR(shm)) {param->u.memref.shm = NULL;return PTR_ERR(shm);}param->u.memref.shm = shm;d->is_supp_ctx = true;/** Pop, own and validate the request in one critical section so that* a concurrent supp_abort_all() can never free the request while* it is being prepared here.*/while (true) {mutex_lock(&supp->mutex);if (supp->shutting_down) {mutex_unlock(&supp->mutex);ret = -ESHUTDOWN;goto err;}req = list_first_entry_or_null(&supp->reqs,struct mbedtee_supp_req, node);if (!req) {mutex_unlock(&supp->mutex);if (wait_for_completion_interruptible(&supp->reqs_c)) {ret = -ERESTARTSYS;goto err;}continue;}list_del_init(&req->node);req->worker = current;req->ctx = ctx;list_add_tail(&req->node, &supp->active_reqs);if (req->size > param->u.memref.size) {/** The supplicant cannot grow its buffer dynamically, so* requeueing would retry this request forever. Complete it* with a transport error and keep this ioctl waiting for the* next request instead of terminating a supplicant worker.*/mbedtee_supp_complete_req(req);mutex_unlock(&supp->mutex);continue;}req->shm = shm;*func = req->func;memcpy(shm->kaddr, req->data, req->size);param->u.memref.size = req->size;mutex_unlock(&supp->mutex);return 0;}+err:
tee_shm_put(shm);param->u.memref.shm = NULL;return ret;+}
+/*
- Called by supplicant to send back the result of a request.
- */
+int mbedtee_supp_send(struct tee_context *ctx, u32 ret, u32 num_params,
struct tee_param *param)+{
struct tee_device *teedev = ctx->teedev;struct mbedtee_device *mbedtee = tee_get_drvdata(teedev);struct mbedtee_supp *supp = &mbedtee->supp;struct mbedtee_supp_req *req;struct mbedtee_supp_req *_req;bool invalid;/** Find and detach the active request belonging to this worker thread* and to this supplicant context. Each worker is identified by its* task_struct pointer, which was recorded when supp_recv() dispatched* the request.*/mutex_lock(&supp->mutex);req = NULL;list_for_each_entry(_req, &supp->active_reqs, node) {if (_req->worker == current && _req->ctx == ctx) {req = _req;list_del_init(&req->node);break;}}mutex_unlock(&supp->mutex);if (!req)return -ENOENT;/** Complete the request even on malformed input: the worker waiting* in mbedtee_supp_handler() must not hang because a broken* supplicant sent invalid parameters.*/invalid = num_params != 1 || !tee_param_is_memref(param) ||param->u.memref.size > req->size;if (invalid) {if (req->data && req->size >= sizeof(struct supp_cmd_hdr))((struct supp_cmd_hdr *)req->data)->ret =TEEC_ERROR_COMMUNICATION;} else {memcpy(req->data, req->shm->kaddr, param->u.memref.size);}if (req->shm)tee_shm_put(req->shm);req->shm = NULL;/* Wake up mbedtee_supp_handler(). */complete(&req->c);return invalid ? -EINVAL : 0;+} diff --git a/include/uapi/linux/tee.h b/include/uapi/linux/tee.h index 5203977..2d30296 100644 --- a/include/uapi/linux/tee.h +++ b/include/uapi/linux/tee.h @@ -60,6 +60,7 @@ #define TEE_IMPL_ID_AMDTEE 2 #define TEE_IMPL_ID_TSTEE 3 #define TEE_IMPL_ID_QTEE 4 +#define TEE_IMPL_ID_MBEDTEE 5
/*
- OP-TEE specific capabilities
-- 2.43.0