Hi Sebastian,

Want to know the scene when the secure fault:

* Are you saying that after couple of rounds of tests a secure fault would occur? * It happens on a specific version, but not the release and the latest master?

There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.

Created task if you want to paste some debug code and data: https://developer.trustedfirmware.org/T975

Thanks.

/Ken

From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Sebastian Bøe via TF-M Sent: Thursday, November 11, 2021 5:56 PM To: tf-m@lists.trustedfirmware.org Subject: [TF-M] race condition in IPC

Hi,

are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?

More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which is 38 commits prior to TF-Mv1.4.0-RC1.

I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.

More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.

This is run on the nrf platform as a part of the nrf SDK.

Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'

It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.

#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()

I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.

Sebastian Bøe

Will try to reproduce in local firstly.

The commit id is: a199c3047f320a2f82b9a0c27af5b50991184e0f, and you are testing with Nordic platform, right? We may try it on other platforms as we may not have the exact boards.

/Ken

From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Ken Liu via TF-M Sent: Thursday, November 11, 2021 6:12 PM To: tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] race condition in IPC

Hi Sebastian,

Want to know the scene when the secure fault:

* Are you saying that after couple of rounds of tests a secure fault would occur? * It happens on a specific version, but not the release and the latest master?

There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.

Created task if you want to paste some debug code and data: https://developer.trustedfirmware.org/T975

Thanks.

/Ken

From: TF-M <tf-m-bounces@lists.trustedfirmware.orgmailto:tf-m-bounces@lists.trustedfirmware.org> On Behalf Of Sebastian Bøe via TF-M Sent: Thursday, November 11, 2021 5:56 PM To: tf-m@lists.trustedfirmware.orgmailto:tf-m@lists.trustedfirmware.org Subject: [TF-M] race condition in IPC

Hi,

are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?

More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which is 38 commits prior to TF-Mv1.4.0-RC1.

I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.

More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.

This is run on the nrf platform as a part of the nrf SDK.

Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'

It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.

#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()

I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.

Sebastian Bøe

Honestly, I wouldn't recommend trying to reproduce, you need to be very lucky with how the code is built, and also be careful not to reboot, as it only triggers on the first run, not after system reset.

I just wanted to make sure I wasn't debugging a known race-condition.

Also, if someone knows how to extract a backtrace from a secure fault handler, then that would also be appreciated. Either via gdb or C.

I would appreciate a link to the dual-core issue, if it is IPC related, then it might be related after all.

Sebastian Bøe

On Thu, 11 Nov 2021 at 11:34, Sebastian Bøe sebastianbooe@gmail.com wrote:

Yes,

the problem is that it is also tested as a part of the nRF SDK, which results in a slightly different build, so it might not reproduce locally. Will try to reproduce without the nRF SDK also.

Sebastian Bøe

On Thu, 11 Nov 2021 at 11:25, Ken Liu via TF-M < tf-m@lists.trustedfirmware.org> wrote:

...

Will try to reproduce in local firstly.

The commit id is: a199c3047f320a2f82b9a0c27af5b50991184e0f, and you are testing with Nordic platform, right? We may try it on other platforms as we may not have the exact boards.

/Ken

*From:* TF-M tf-m-bounces@lists.trustedfirmware.org * On Behalf Of *Ken Liu via TF-M *Sent:* Thursday, November 11, 2021 6:12 PM *To:* tf-m@lists.trustedfirmware.org *Cc:* nd nd@arm.com *Subject:* Re: [TF-M] race condition in IPC

Hi Sebastian,

Want to know the scene when the secure fault:

• Are you saying that after couple of rounds of tests a secure fault

would occur?

• It happens on a specific version, but not the release and the

latest master?

There is one race condition problem under debugging for dual-core based platforms, and it looks not the case you are testing.

Created task if you want to paste some debug code and data:

https://developer.trustedfirmware.org/T975

Thanks.

/Ken

*From:* TF-M tf-m-bounces@lists.trustedfirmware.org *On Behalf Of *Sebastian Bøe via TF-M *Sent:* Thursday, November 11, 2021 5:56 PM *To:* tf-m@lists.trustedfirmware.org *Subject:* [TF-M] race condition in IPC

Hi,

are there any known race conditions in IPC/PS that affect TF-M 1.4-ish?

More specifically TF-M revision a199c3047f320a2f82b9a0c27af5b50991184e0f, which

is 38 commits prior to TF-Mv1.4.0-RC1.

I am observing that adding printfs, or otherwise changing the flash alignment and therefore runtime of my code will affect whether the PS reliability test suite passes or not.

More specifically, if it triggers a secure fault after x iterations of 2001, or y iterations of 2002, etc. depending on where I put my printfs or otherwise affect the alignment of the code.

This is run on the nrf platform as a part of the nrf SDK.

Test Suite PS reliability tests (TFM_PS_TEST_2XXX)... 'repetitive sets and gets in/from an asset' 'repetitive sets, gets and removes'

It does not reproduce under GDB and I am unable to unwind the backtrace from the secure fault handler. Any tips about recovering a backtrace from a secure fault handler would also be appreciated.

#0 SecureFault_Handler () at /home/sebo/ncs/modules/tee/tfm/trusted-firmware-m/secure_fw/spm/cmsis_psa/arch/tfm_arch_v8m_main.c:96 #1 0xffffffb4 in ?? ()

I have not been able to reproduce it in the latest upstream TF-M release/revision, but that could just be because I haven't been able to hit the race condition.

Sebastian Bøe

TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m