I don't know that the SPE needs to be restored to its IDLE state prior to a thread pre-empting NS interrupt occurring. I just assumed that these 2 rules required it:
a. All of the SPE interrupts must have higher priority than NSPE interrupts d. A NSPE interrupt is not allowed to preempt SPE ISR (I assumed that this rule applies to the SP while it is processing the ISR as well).
Alan
-----Original Message----- From: TF-M [mailto:tf-m-bounces@lists.trustedfirmware.org] On Behalf Of Edison Ai (Arm Technology China) via TF-M Sent: Thursday, November 14, 2019 12:58 AM To: tf-m@lists.trustedfirmware.org Cc: nd Subject: Re: [TF-M] [EXTERNAL] RE: secure interrupt handlng in IPC mode
Hi Alan,
I have one question, why the SPE needs restored to its IDLE state prior to a thread pre-empting NS interrupt occurring?
About a secure interrupt is configured with a lower priority than pendsv, I do not think it is a good idea. The pendsv is used to trigger the scheduler to switch thread. The normal interrupt process should like this: Interrupt happen -> jump into related ISR -> do necessary process such as mask interrupt status, send even or signal to related SP. -> set pendsv to trigger scheduler -> jump out ISR -> scheduler to decide which thread to run. In these sequences, if there are many interrupts happen at the same time, and high priority interrupt will interrupt low priority interrupt. Each ISR will trigger pendsv. But only one pendsv hander function needs to be processed.
If we set the pendsv with high priority. Each interrupt will trigger pendsv and the pendsv handler will be processed every time. It will be more complex and spend more CPU time.
Sorry that I do not know what is your detail scenario, and it will be more clear and helpful if you can provide more information.
Thanks, Edison
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Ken Liu (Arm Technology China) via TF-M Sent: Thursday, November 14, 2019 3:12 PM To: tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] [EXTERNAL] RE: secure interrupt handlng in IPC mode
Hi Alan,
I have created the cooperative scheduling doc link here for comment: https://review.trustedfirmware.org/c/trusted-firmware-m/+/2466
BR
/Ken
-----Original Message----- From: DeMars, Alan ademars@ti.com Sent: Thursday, November 14, 2019 10:44 AM To: Mate Toth-Pal Mate.Toth-Pal@arm.com Cc: tf-m@lists.trustedfirmware.org; nd nd@arm.com; Ken Liu (Arm Technology China) Ken.Liu@arm.com Subject: Re: [EXTERNAL] RE: secure interrupt handlng in IPC mode
Mate,
Apart from the surprising behavior actually working to my benefit, my concern is the implications of violating the Cooperative Scheduling Rules I quoted below. The normal behavior (ie the SP running in thread mode with no exception priority when servicing a secure interrupt) seems to violate those rules. Is this a problem for the scheduler?
Alan
On Nov 13, 2019, at 5:08 AM, Mate Toth-Pal Mate.Toth-Pal@arm.com wrote:
Hi Alan,
That's a quite difficult question I think.
At first look, it seems a to be a valid thinking that a secure partition that is handling an IRQ signal should be considered as an SPE ISR.
However implementing this seems to be a nontrivial task.
The code in the secure partitions is organised into a single thread, that executes a while loop, calls 'psa_wait()' in every iteration, and acts on the signals returned by it.
The phenomenon we observed with your setup was the following:
- When the Handler mode interrupt handler, that had lower priority than the PendSV reenabled interrupts after setting the signal, was interrupted by the PendSV handler.
- PendSV handler scheduled the partition with the interrupt signal set, and did an exception return to thread mode.
- The code of the Secure partition thread started executing, in Thread mode, but with the priority of the interrupt from the first point, as that interrupt handler was interrupted before it could do exception return to acknowledge the handling of the interrupt in the NVIC.
- After the partition thread code called psa_wait again, the scheduler restored the context of the interrupt handler from the first point, that handler did an exception return, and everything is 'back to normal'
Now, if I understand properly, your proposal is to make the conditions in point 3. to be mandatory for any scheduler implementation.
To make this happen in TF-M, all the secure interrupts should have lower priority than the PendSV handler. This is necessary so that the above detailed behaviour is valid for all the interrupts.
The problem with this is that the SPM has no control over which signal the code of the secure partition is going to execute. So in the current design a secure partition might decide to handle a service call signal (if there is any) instead of an IRQ signal, and that would be against our intention. We might be able to modify psa_wait to return only IRQ signals if there is any active, but that would be a violation of the PSA FF spec.
Regards, Mate
From: DeMars, Alan ademars@ti.com Sent: 12 November 2019 17:47 To: Mate Toth-Pal Mate.Toth-Pal@arm.com Cc: Ken Liu (Arm Technology China) Ken.Liu@arm.com Subject: RE: secure interrupt handlng in IPC mode
You can CC the tf-m mailing list.
From: Mate Toth-Pal [mailto:Mate.Toth-Pal@arm.com] Sent: Tuesday, November 12, 2019 8:02 AM To: DeMars, Alan Cc: Ken Liu (Arm Technology China) Subject: [EXTERNAL] RE: secure interrupt handlng in IPC mode
Hi Alan
Would it be OK to you if I cc the TF-M mailing list in my answer?
Or if you would prefer that, I can move the 'Cooperative Scheduling Rules' document to the TF-M source (in the docs folder), and you can create a patch for it in gerrit.
Thanks, Mate
From: DeMars, Alan mailto:ademars@ti.com Sent: 11 November 2019 21:05 To: Mate Toth-Pal mailto:Mate.Toth-Pal@arm.com Cc: Ken Liu (Arm Technology China) mailto:Ken.Liu@arm.com Subject: secure interrupt handlng in IPC mode
Hi Mate!
I've been doing some thinking about the unexpected behavior we saw when a secure interrupt is configured with a lower priority than pendsv.
I think the below statements in the Cooperative Scheduling Rules discussion:
a. All of the SPE interrupts must have higher priority than NSPE interrupts d. A NSPE interrupt is not allowed to preempt SPE ISR must extend to the priority level of the SP as it services the interrupt while in thread mode.
Otherwise the problem that these rules are meant to avoid can arise: the SPE is not completely restored to its IDLE state prior to a thread pre-empting NS interrupt occurring.
So, rather than the behavior we witnessed being unexpected, I think it may have to be enforced.
Is this correct thinking?
Alan
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Hi Alan,
Thanks for your information.
So I think there are 2 discussion here: 1. You had met unexpected behavior when a secure interrupt is configured with a lower priority than pendsv. As I understand, if you just move the pendsv in a higher priority, it should be ok. Just as I say in former mail that the pendsv handler will be processed every time, but it still can works. In this case, the 'd' point you list should not extend to SP. 2. You say that "the SP while it is processing the ISR". Do you mean you want to add a SP ISR? Right? please correct me if it is not. If so the current discussion should be if it is allowed to process ISR in SP: - If allow, "A NSPE interrupt is not allowed to preempt SPE ISR" could be understood as a NSPE interrupt is not allowed to preempt SP ISR. So I think this why you assume the SPE needs to be restored to the IDLE state. - If no. you assume will not reasonable. I think the answer should be NO. PSA FF gives us several APIs to process the interrupt: psa_wait, psa_get, psa_eoi based on the message in the IPC model. In this mechanism, ISR needs to send a signal to SP and let SP process data after scheduling. It is cleaner and security.
Thanks, Edison
-----Original Message----- From: DeMars, Alan ademars@ti.com Sent: Friday, November 15, 2019 4:48 AM To: Edison Ai (Arm Technology China) Edison.Ai@arm.com Cc: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org Subject: RE: [EXTERNAL] RE: secure interrupt handlng in IPC mode
I don't know that the SPE needs to be restored to its IDLE state prior to a thread pre-empting NS interrupt occurring. I just assumed that these 2 rules required it:
a. All of the SPE interrupts must have higher priority than NSPE interrupts d. A NSPE interrupt is not allowed to preempt SPE ISR (I assumed that this rule applies to the SP while it is processing the ISR as well).
Alan
-----Original Message----- From: TF-M [mailto:tf-m-bounces@lists.trustedfirmware.org] On Behalf Of Edison Ai (Arm Technology China) via TF-M Sent: Thursday, November 14, 2019 12:58 AM To: tf-m@lists.trustedfirmware.org Cc: nd Subject: Re: [TF-M] [EXTERNAL] RE: secure interrupt handlng in IPC mode
Hi Alan,
I have one question, why the SPE needs restored to its IDLE state prior to a thread pre-empting NS interrupt occurring?
About a secure interrupt is configured with a lower priority than pendsv, I do not think it is a good idea. The pendsv is used to trigger the scheduler to switch thread. The normal interrupt process should like this: Interrupt happen -> jump into related ISR -> do necessary process such as mask interrupt status, send even or signal to related SP. -> set pendsv to trigger scheduler -> jump out ISR -> scheduler to decide which thread to run. In these sequences, if there are many interrupts happen at the same time, and high priority interrupt will interrupt low priority interrupt. Each ISR will trigger pendsv. But only one pendsv hander function needs to be processed.
If we set the pendsv with high priority. Each interrupt will trigger pendsv and the pendsv handler will be processed every time. It will be more complex and spend more CPU time.
Sorry that I do not know what is your detail scenario, and it will be more clear and helpful if you can provide more information.
Thanks, Edison
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Ken Liu (Arm Technology China) via TF-M Sent: Thursday, November 14, 2019 3:12 PM To: tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] [EXTERNAL] RE: secure interrupt handlng in IPC mode
Hi Alan,
I have created the cooperative scheduling doc link here for comment: https://review.trustedfirmware.org/c/trusted-firmware-m/+/2466
BR
/Ken
-----Original Message----- From: DeMars, Alan ademars@ti.com Sent: Thursday, November 14, 2019 10:44 AM To: Mate Toth-Pal Mate.Toth-Pal@arm.com Cc: tf-m@lists.trustedfirmware.org; nd nd@arm.com; Ken Liu (Arm Technology China) Ken.Liu@arm.com Subject: Re: [EXTERNAL] RE: secure interrupt handlng in IPC mode
Mate,
Apart from the surprising behavior actually working to my benefit, my concern is the implications of violating the Cooperative Scheduling Rules I quoted below. The normal behavior (ie the SP running in thread mode with no exception priority when servicing a secure interrupt) seems to violate those rules. Is this a problem for the scheduler?
Alan
On Nov 13, 2019, at 5:08 AM, Mate Toth-Pal Mate.Toth-Pal@arm.com wrote:
Hi Alan,
That's a quite difficult question I think.
At first look, it seems a to be a valid thinking that a secure partition that is handling an IRQ signal should be considered as an SPE ISR.
However implementing this seems to be a nontrivial task.
The code in the secure partitions is organised into a single thread, that executes a while loop, calls 'psa_wait()' in every iteration, and acts on the signals returned by it.
The phenomenon we observed with your setup was the following:
- When the Handler mode interrupt handler, that had lower priority than the PendSV reenabled interrupts after setting the signal, was interrupted by the PendSV handler.
- PendSV handler scheduled the partition with the interrupt signal set, and did an exception return to thread mode.
- The code of the Secure partition thread started executing, in Thread mode, but with the priority of the interrupt from the first point, as that interrupt handler was interrupted before it could do exception return to acknowledge the handling of the interrupt in the NVIC.
- After the partition thread code called psa_wait again, the scheduler restored the context of the interrupt handler from the first point, that handler did an exception return, and everything is 'back to normal'
Now, if I understand properly, your proposal is to make the conditions in point 3. to be mandatory for any scheduler implementation.
To make this happen in TF-M, all the secure interrupts should have lower priority than the PendSV handler. This is necessary so that the above detailed behaviour is valid for all the interrupts.
The problem with this is that the SPM has no control over which signal the code of the secure partition is going to execute. So in the current design a secure partition might decide to handle a service call signal (if there is any) instead of an IRQ signal, and that would be against our intention. We might be able to modify psa_wait to return only IRQ signals if there is any active, but that would be a violation of the PSA FF spec.
Regards, Mate
From: DeMars, Alan ademars@ti.com Sent: 12 November 2019 17:47 To: Mate Toth-Pal Mate.Toth-Pal@arm.com Cc: Ken Liu (Arm Technology China) Ken.Liu@arm.com Subject: RE: secure interrupt handlng in IPC mode
You can CC the tf-m mailing list.
From: Mate Toth-Pal [mailto:Mate.Toth-Pal@arm.com] Sent: Tuesday, November 12, 2019 8:02 AM To: DeMars, Alan Cc: Ken Liu (Arm Technology China) Subject: [EXTERNAL] RE: secure interrupt handlng in IPC mode
Hi Alan
Would it be OK to you if I cc the TF-M mailing list in my answer?
Or if you would prefer that, I can move the 'Cooperative Scheduling Rules' document to the TF-M source (in the docs folder), and you can create a patch for it in gerrit.
Thanks, Mate
From: DeMars, Alan mailto:ademars@ti.com Sent: 11 November 2019 21:05 To: Mate Toth-Pal mailto:Mate.Toth-Pal@arm.com Cc: Ken Liu (Arm Technology China) mailto:Ken.Liu@arm.com Subject: secure interrupt handlng in IPC mode
Hi Mate!
I've been doing some thinking about the unexpected behavior we saw when a secure interrupt is configured with a lower priority than pendsv.
I think the below statements in the Cooperative Scheduling Rules discussion:
a. All of the SPE interrupts must have higher priority than NSPE interrupts d. A NSPE interrupt is not allowed to preempt SPE ISR must extend to the priority level of the SP as it services the interrupt while in thread mode.
Otherwise the problem that these rules are meant to avoid can arise: the SPE is not completely restored to its IDLE state prior to a thread pre-empting NS interrupt occurring.
So, rather than the behavior we witnessed being unexpected, I think it may have to be enforced.
Is this correct thinking?
Alan
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m
tf-m@lists.trustedfirmware.org
-
DeMars, Alan -
Edison Ai (Arm Technology China)