Hi Chirs,
I understand your concern about IPC in synchronization now. I think it can be very natural that different IPC resources are used for synchronization and mailbox services if vendor selects IPC as synchronization mechanism. The implementations can be also different. The IPC implementation used in synchronization can be very simple. The initiator can just toggle a status bit in a IPC channel. The peer core loops the status bit. It is the current implementation in our PoC. Even if the interrupt is used in synchronization, I think it is a very basic requirement that the IRQ should be masked until the driver is ready. So the IPC interrupt will be pending until the peer core enters the synchronization step.
However, I believe that it is unncessary to discuss about whether IPC can be used in synchronization in the design document. We can directly recommend shared memory and allow others to use their own favorate mechanism. IMO, maybe we don't even need to mention IPC for the synchronization.
For `tfm_spm_hal_wait_for_s_cpu_ready()`, we can use a compile flag to enable/disable it in dual core scenario/singl core scenario during building. There should be a lot of functions in the same case. We can discuss in detail with Danny in the build system design
Thank you.
Best regards, Hu Ziji
On 3/29/2019 4:49 AM, Christopher Brand wrote:
Hi David,
-----Original Message----- From: David Hu (Arm Technology China) David.Hu@arm.com Sent: Thursday, March 28, 2019 3:37 AM To: Christopher Brand chris.brand@cypress.com; tf- m@lists.trustedfirmware.org Cc: Danny Shavit Danny.Shavit@arm.com; nd nd@arm.com Subject: Re: [RFC] twin cpu bootloader design document
Hi Chris,
Sorry that the format in previous reply is broken. Re-send the reply. Please check my inline comments in below.
On 3/27/2019 1:44 AM, Christopher Brand wrote:
Hi David,
-----Original Message----- From: David Hu (Arm Technology China) David.Hu@arm.com Sent: Tuesday, March 26, 2019 12:06 AM To: tf-m@lists.trustedfirmware.org; Christopher Brand chris.brand@cypress.com Cc: Danny Shavit Danny.Shavit@arm.com; nd nd@arm.com Subject: Re: [RFC] twin cpu bootloader design document
Hi Chris,
Sorry for the delayed reply. Please check my comments in below. Just about small details. Please correct me if I misunderstand anything.
- In my own opinion, it can be possible to use IPC to synchronize
mailbox services between two cores, during initialization.
It certainly *may* be possible, but I'm not convinced that it's *always*
possible. I mean if the synchronization is used to make sure the IPC/mailbox mechanism is ready, then the IPC should be ready when synchronization starts. Any corner case?
No, IPC is only guaranteed to be ready when synchronization *ends*, not starts. It will be ready when the *second* CPU calls the synchronization function, not when the *first* does.
Both sides need to do any IPC setup before calling the synchronization function, but that's asynchronous between the two CPUs - just because the non-secure CPU has finished doing whatever IPC setup is needed on its side doesn't mean that the secure CPU has also reached that point.
For example, let's assume that both sides need to install an IPC interrupt handler, and the secure CPU also has to write some values to shared memory to initialize a ring buffer. You could have this sequence:
- Non-secure CPU installs IPC interrupt handler
- Secure CPU writes to shared memory to set up ring buffer
- Non-secure CPU calls synchronization function
- Secure CPU installs IPC interrupt handler
- Secure CPU calls synchronization function
Clearly, generating an IPC interrupt to the secure CPU at point 3 could cause problems, because the handler hasn't yet been installed.
On the NS CPU, the synchronization point means that the secure CPU has done all the things that it does in a single CPU system before jumping to the NS code. On the S CPU, the synchronization point means that the non-secure CPU has done all the things it needs to do before starting the code that would be jumped to in a single-CPU system. It's not just about the IPC mechanism.
In the current, single-CPU TF-M system, there's an implicit "synchronization point" because the secure code makes a call to start the non-secure code. I think you prevent possible problems due to unfulfilled assumptions if you retain that synchrnoisation point when you have two CPUs that each have their own initialization to do.
The synchronization is only trigged when the mailbox mechanismis ready on mailbox server or client. It means that the IPC module should be also configured.
If the non-secure side is particularly fast, it could attempt to use the IPC mechanism before the secure side has finished setting it up, which
could lead to problems. Yes and the attempt will be blocked until the mailbox/IPC mechanism is ready on peer core.
Blocked how?
We definitely can implement shared memory in TF-M as a reference. May I suggest that it would be more reasonable to allow diverse mechasims to implement the synchronization?
Yes, the method of synchronization is intended to be platform-specific - as it says in the document "Both the secure and non-secure code will make platform-specific calls to establish a synchronization point".
As long as we keep the API(s) and sequence generic enough, platform vendors can implement thier own optimizations.
Absolutely. That's the intent.
Based on the above assumption, using IPC to synchronize the twocores is more generic and convenient than accessing shared memory.
It's platform-specific code, so each platform can use the method that works best there, that could be IPC for some platforms, but I'm not convinced that we can rely on using IPC on platforms where the NS CPU is
much faster than the secure CPU.
- I aggree that IPC might not be always available on every platform while
some platform vendors may prefer it. So I wonder if we can keep shared memory as the default choice and implement it as a reference implementation in TF-M. Besides, we can add a comment that platform vendor can implement their own synchronization besides default shared memory mechanism. 2. IMO, it is irrelevant to the speed/performance of two cores. No matter whether NS CPU is faster, slower, or at the same speed, the synchronization for mailbox communication is always essential. Even if secure CPU is more powerful, NS CPU still needs to check whether mailbox is ready in secure side. ( I guess it is a little off the topic.)
I think it is on-topic. With the synchronization point calls, both sides have a guarantee that the mailbox is ready after it has made that call, so they shouldn't need to make any further checks.
If using shared memory to pass status flag, it can be necessaryto adjust the address of the shared memory occasionally according to the memory assignment in different applications.
The shared memory would only be needed during early initialization. As soon as both sides have passed that point, the memory could be re-assigned for any other use. The memory would always be available for other uses before the NS OS has started running.
I aggree that the shared memory for synchronization can re-used as other init module. Actually I care about more about the details before the shared memory is used, such as 1. The address is defined in secure side and has to be exported to NS side. How does NS side code import the address? 2. The value in shared memory address should be initialized by SPE side before the NSPE side kicks off 3. The shared memory should be put in a privileged region in NSPE. However, it is a little difficult for SPE to know the region assignment in NSPE. Do you think if we can specify more requirements on the shared memory implementations? It will help enhance the security of our developement.
I think it's all platform-specific. From a design point of view, the requirement is simply "neither core can exit the synchronization functions until both have entered them". How that is implemented is up to the system designer - shared memory, semaphores, IPC, bits in registers, whatever...
- I'd like to suggest that we shall discuss more about when the
booting HAL APIs are invoked in TF-M. `tfm_core_init()` initializes the TF-M core. Thus in theory, `tfm_core_init()` is irrelevant to the system topology or platform
implementations.
As a result, IMO, it can be more reasonable to put the HAL APIsoutside the `tfm_core_init()`.
This could well be a misunderstanding of the terminology on my part (I'm
fairly new here 😊).
Does "core" here mean "CPU"? I was reading it as "the core part of TFM", which then would include the parts on both CPUs. I note that tfm_core_init() already makes a call to configure_ns_code(), which is not
part of initializing the secure CPU...
Moving this new call from within tfm_core_init() to main() immediately after the call to tfm_core_init() doesn't look like it would make much difference from a performance perspective.
Sorry for misleading you :) In my very own opinion, since the topology concept is brought into TF-M, the init sequence might be sightly adjusted. We can adjust timing of calling APIs if we find a more appropriate place.
Sure. This is probably easier to figure out when there are patches to look at.
- ` tfm_spm_hal_wait_for_ns_cpu_ready()` can be optional.
Right. As the final sentence of the document states, my intent is to allow platforms to not define any or all of the three new functions as
needed.
The secure core acts as a server and it is driven by the requestfrom NS core. The secure core actually doesn't have to wait for an explicit signal to know NS is ready.
My concern is that I want to be as flexible as possible regarding platform architecture.
The synchronization can be guaranteed if NS core starts requestvia mailbox only after secure core is available.
Definitely true for some implementations. I'm not confident that I can say that it's true for all implementations on all platforms.
I guess there are two ways to go - we can include this call now and remove it after we see that there are several twin CPU platforms and none of them need it, or we can exclude it now and add it when and if a
platform needs it. Yep. We can keep it and just comment that it is not a must.
Ok
- It can be unnecessary to require calling
`tfm_spm_hal_wait_for_s_cpu_ready()` in NS `main()`. It might be too early to wait in `main()` and may block other initializations which don't rely on mailbox.
We definitely want to make this call as late as possible in NS main(). In the existing TF-M code, I don't think we can assume that the call to osKernelInitialise() from NS main() won't ever use any secure services, so we need to know that the secure code is ready to provide those services. If that's an acceptable constraint then we can move this call later.
Conceptually at least, this call could just record that the secure side is not yet ready and that the check needs to be made again before trying to communicate with it, with corresponding code in the mailbox/IPC code to make the check again (and to wait if needed) in that
case. That would be a platform-specific optimization.
Note, though, that doing so would mean that any shared memory would
be
tied up for longer. It really does feel like the kind of decision that has to be made by the system designer. My goal with the design document is to specify a system that is simple and guaranteed to work while allowing the system designer to optimize it for their system as
needed. Similary as above, I'd like to suggest that we can implement a reference design which puts the synchonrization in main() by default. In the meantime, I wonder if we can make the statement about sequence more flexible that different implementations can be allowed. Some requirments and limitiations can be added.
Yes, that makes sense. There is presumably a lot of system-specific code that gets executed later on the NS CPU, anyway.
This API can be invoked in mailbox functionalities. The whole NSinitialization can continue, including enabling application threads, until a NS application requests Secure services via mailbox at the very first
time.
Again, my concern is that the NS side could boot so fast that it attempts to use the mailbox/IPC mechanism before the secure side has set it up - you don't want the NS side generating an IPC interrupt before the secure side has installed a handler for that interrupt, for
example.
Thus the whole dual core design can be more generic since themailbox workflow should be identical on diverse platforms. And we can save the time and effort to hack each RTOS initialization. In other words, I wonder if we can make calling `tfm_spm_hal_wait_for_s_cpu_ready()` in NS `main() as an option and allow other implementations.
Again, the document states that none of these new functions need to be provided If they're not needed on the platform.
Actually, in contrast, I prefer to mark this function as mandatory to protect SPE from being disturbed by NSPE. 😝
I don't want to require it on a single-CPU system, though...
Chris
Thank you.
Best regards, Hu Ziji
Date: Thu, 14 Mar 2019 18:50:56 +0000 From: Christopher Brand chris.brand@cypress.com To: "tf-m@lists.trustedfirmware.org" tf-m@lists.trustedfirmware.org Subject: [TF-M] [RFC] twin cpu bootloader design document Message-ID: <BYAPR06MB5301EBF02F4C0B60A9BB7742FE4B0@BYAPR06MB5301.n amprd06.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Hi,
I've posted a design document for bootloader changes to support twin cpu at https://developer.trustedfirmware.org/w/tf_m/design/twin- cpu/bootloader/ Comments appreciated!
Thanks,
Chris
This message and any attachments may contain confidential information
from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.
This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.