Hi,
are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?
More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which is 38 commits prior to TF-Mv1.4.0-RC1.
I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.
More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.
This is run on the nrf platform as a part of the nrf SDK.
Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'
It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.
#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()
I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.
Sebastian Bøe
Hi Sebastian,
Want to know the scene when the secure fault:
* Are you saying that after couple of rounds of tests a secure fault would occur? * It happens on a specific version, but not the release and the latest master?
There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.
Created task if you want to paste some debug code and data: https://developer.trustedfirmware.org/T975
Thanks.
/Ken
From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Sebastian Bøe via TF-M Sent: Thursday, November 11, 2021 5:56 PM To: tf-m@lists.trustedfirmware.org Subject: [TF-M] race condition in IPC
Hi,
are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?
More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which is 38 commits prior to TF-Mv1.4.0-RC1.
I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.
More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.
This is run on the nrf platform as a part of the nrf SDK.
Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'
It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.
#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()
I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.
Sebastian Bøe
It happens consistently with the same firmware (and when GDB is not used). But it triggers on different iterations of TFM_S_PS_TEST_2001 and TFM_S_PS_TEST_2002 depending on code alignment or where printfs are placed (which affects code alignment).
And for most code alignments it doesn't trigger at all.
It does not trigger for 1.4 release or master, but that may or may not be because I haven't found the exact alignment that will trigger it.
Sebastian Bøe
On Thu, 11 Nov 2021 at 11:12, Ken Liu via TF-M < tf-m@lists.trustedfirmware.org> wrote:
Hi Sebastian,
Want to know the scene when the secure fault:
- Are you saying that after couple of rounds of tests a secure fault
would occur?
- It happens on a specific version, but not the release and the latest
master?
There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.
Created task if you want to paste some debug code and data:
https://developer.trustedfirmware.org/T975
Thanks.
/Ken
*From:* TF-M tf-m-bounces@lists.trustedfirmware.org * On Behalf Of *Sebastian Bøe via TF-M *Sent:* Thursday, November 11, 2021 5:56 PM *To:* tf-m@lists.trustedfirmware.org *Subject:* [TF-M] race condition in IPC
Hi,
are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?
More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which
is 38 commits prior to TF-Mv1.4.0-RC1.
I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.
More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.
This is run on the nrf platform as a part of the nrf SDK.
Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'
It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.
#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()
I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.
Sebastian Bøe
TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Will try to reproduce in local firstly.
The commit id is: a199c3047f320a2f82b9a0c27af5b50991184e0f, and you are testing with Nordic platform, right? We may try it on other platforms as we may not have the exact boards.
/Ken
From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Ken Liu via TF-M Sent: Thursday, November 11, 2021 6:12 PM To: tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] race condition in IPC
Hi Sebastian,
Want to know the scene when the secure fault:
* Are you saying that after couple of rounds of tests a secure fault would occur? * It happens on a specific version, but not the release and the latest master?
There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.
Created task if you want to paste some debug code and data: https://developer.trustedfirmware.org/T975
Thanks.
/Ken
From: TF-M <tf-m-bounces@lists.trustedfirmware.orgmailto:tf-m-bounces@lists.trustedfirmware.org> On Behalf Of Sebastian Bøe via TF-M Sent: Thursday, November 11, 2021 5:56 PM To: tf-m@lists.trustedfirmware.orgmailto:tf-m@lists.trustedfirmware.org Subject: [TF-M] race condition in IPC
Hi,
are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?
More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which is 38 commits prior to TF-Mv1.4.0-RC1.
I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.
More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.
This is run on the nrf platform as a part of the nrf SDK.
Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'
It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.
#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()
I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.
Sebastian Bøe
Yes,
the problem is that it is also tested as a part of the nRF SDK, which results in a slightly different build, so it might not reproduce locally. Will try to reproduce without the nRF SDK also.
Sebastian Bøe
On Thu, 11 Nov 2021 at 11:25, Ken Liu via TF-M < tf-m@lists.trustedfirmware.org> wrote:
Will try to reproduce in local firstly.
The commit id is: a199c3047f320a2f82b9a0c27af5b50991184e0f, and you are testing with Nordic platform, right? We may try it on other platforms as we may not have the exact boards.
/Ken
*From:* TF-M tf-m-bounces@lists.trustedfirmware.org * On Behalf Of *Ken Liu via TF-M *Sent:* Thursday, November 11, 2021 6:12 PM *To:* tf-m@lists.trustedfirmware.org *Cc:* nd nd@arm.com *Subject:* Re: [TF-M] race condition in IPC
Hi Sebastian,
Want to know the scene when the secure fault:
- Are you saying that after couple of rounds of tests a secure fault
would occur?
- It happens on a specific version, but not the release and the latest
master?
There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.
Created task if you want to paste some debug code and data:
https://developer.trustedfirmware.org/T975
Thanks.
/Ken
*From:* TF-M tf-m-bounces@lists.trustedfirmware.org *On Behalf Of *Sebastian Bøe via TF-M *Sent:* Thursday, November 11, 2021 5:56 PM *To:* tf-m@lists.trustedfirmware.org *Subject:* [TF-M] race condition in IPC
Hi,
are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?
More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which
is 38 commits prior to TF-Mv1.4.0-RC1.
I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.
More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.
This is run on the nrf platform as a part of the nrf SDK.
Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'
It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.
#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()
I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.
Sebastian Bøe
TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Honestly, I wouldn't recommend trying to reproduce, you need to be very lucky with how the code is built, and also be careful not to reboot, as it only triggers on the first run, not after system reset.
I just wanted to make sure I wasn't debugging a known race-condition.
Also, if someone knows how to extract a backtrace from a secure fault handler, then that would also be appreciated. Either via gdb or C.
I would appreciate a link to the dual-core issue, if it is IPC related, then it might be related after all.
Sebastian Bøe
On Thu, 11 Nov 2021 at 11:34, Sebastian Bøe sebastianbooe@gmail.com wrote:
Yes,
the problem is that it is also tested as a part of the nRF SDK, which results in a slightly different build, so it might not reproduce locally. Will try to reproduce without the nRF SDK also.
Sebastian Bøe
On Thu, 11 Nov 2021 at 11:25, Ken Liu via TF-M < tf-m@lists.trustedfirmware.org> wrote:
Will try to reproduce in local firstly.
The commit id is: a199c3047f320a2f82b9a0c27af5b50991184e0f, and you are testing with Nordic platform, right? We may try it on other platforms as we may not have the exact boards.
/Ken
*From:* TF-M tf-m-bounces@lists.trustedfirmware.org * On Behalf Of *Ken Liu via TF-M *Sent:* Thursday, November 11, 2021 6:12 PM *To:* tf-m@lists.trustedfirmware.org *Cc:* nd nd@arm.com *Subject:* Re: [TF-M] race condition in IPC
Hi Sebastian,
Want to know the scene when the secure fault:
- Are you saying that after couple of rounds of tests a secure fault
would occur?
- It happens on a specific version, but not the release and the
latest master?
There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.
Created task if you want to paste some debug code and data:
https://developer.trustedfirmware.org/T975
Thanks.
/Ken
*From:* TF-M tf-m-bounces@lists.trustedfirmware.org *On Behalf Of *Sebastian Bøe via TF-M *Sent:* Thursday, November 11, 2021 5:56 PM *To:* tf-m@lists.trustedfirmware.org *Subject:* [TF-M] race condition in IPC
Hi,
are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?
More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which
is 38 commits prior to TF-Mv1.4.0-RC1.
I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.
More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.
This is run on the nrf platform as a part of the nrf SDK.
Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'
It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.
#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()
I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.
Sebastian Bøe
TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m
tf-m@lists.trustedfirmware.org
-
Ken Liu -
Sebastian Bøe