Various issues in successfully using GDB to debug secure apps on Musca S1 connected over USB DAPLink to a Linux host
Now sure where exactly this belongs, but I already posted question abut the Musca S1 earlier, and this seems relevant. ARM support essentially seem to tell me that they cannot support GDB and they are not certain of DAPLink, and recommend ULINKPLUS debugger with ARM IDE. So wanted to get a sense from anyone who has attempted to use this.
In the following I describe the steps to obtain a base line, then provide description in what way GDB / DAPLink seems to misbehave.
To get a base line: ~~~~~~~~~~~~~ - I build 'samples/hello_world' app using Zephyr 'v3.4.0'. Copy the 'build/zephyr/zephyr.hex' file to '/media/user/MUSCA_S1' directory when the board is USB. Connect a terminal emulator to the UART exposed by USB DAPLink, e.g. 'picocom -b 115200 -y n /dev/ttyACM0'. I can see on the console the followinbg *** Booting Zephyr OS build zephyr-v3.4.0 *** Hello World! musca_s1 If I press the reset button, the app restarts as well. This is as expected. I note that this is entirely running as in secure mode, that it does not use TF-M.
- Next I start 'pyocd-gdbserver'. Then start a GDB session using the 'zephyr.elf' file, 'target remote localhost:3333' to connect it to the gdbserver. It connects ok. So I put a breakpoint at "main", do a 'monitor reset halt', and then continue. The breakpoint at 'main' is hit, I can obtain the backtrace to see it is correct as expected.
So my setup seems good.
Next I attempt to test a non-secure app with TF-M. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Build the app using Zephyr build env 'west build -b v2m_musca_s1_ns samples/tfm_integration/tfm_psa_test/ -DCONFIG_TFM_PSA_TEST_CRYPTO=y'. Copy 'build/zephyr/tfm_merged.hex' to '/media/user/MUSCA_S1'. Connect a terminal emulator, picocom, to the console, and hit the PBON (power on) button. I see the following output as expected on the console:
Musca-S1 Dual Firmware Version 1.9 [WRN] This device was provisioned with dummy keys. This device is NOT SECURE [Sec Thread] Secure image initializing! Booting TF-M 79a6115d3 [INF][Crypto] Initialising HW accelerator... complete. *** Booting Zephyr OS build zephyr-v3.4.0 *** ***** PSA Architecture Test Suite - Version 1.4 ***** Running.. Crypto Suite ****************************************** TEST: 201 | DESCRIPTION: Testing crypto key management APIs | UT: psa_crypto_init [Info] Executing tests from non-secure
This is as expected.
However, if I press the reset button the board does not appear to reset - this is unexpected. Has anyone seen a different behavior? Is the specific board that I have is misbehaving?
- Now I start PyOCD GDB server, start GDB and successfully connect it to the GDB server. The target is halted where it was executing, e.g. Remote debugging using localhost:3333 CC_HalWaitInterruptRND (data=data@entry=1024) at /home/zephyr/zephyrproject/modules/tee/tf-m/trusted-firmware-m/lib/ext/cryptocell-312-runtime/host/src/hal/cc3x/cc_hal.c:107 107 irr = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IRR)); (gdb)
This is as expected.
- Next I put a breakpoint at 'sprt_main', then issue a 'monitor reset halt', then issue a 'continue',
The breakpoint is is NOT hit. I see a blue LED blink at perhaps 4 times per second.
This is unexpected. Has anyone seen this behave correctly? Do I have a board that is misbehaving?
Adding the the last part of my previous message:
When I issue GDB the 'continue' command, nothing is output on the console.
As an additional effort to work around the potential issue, I instrument 'runtime_init_c' (called by 'sprt_main') to have the code spin indefinitely waiting for a volatile unsigned int variable 'gdbSpinHere' varaiable to be changed from the default value of 0xffffffff to 0x0. Using a new debug session, when I issue GDB 'target remote localhost:3333', GDB connects to the PyOCD GDBServer. The backtrace shows 'runtime_init_c' is on the top of the stack. This is as expected.
Next I update 'gdbSpinHere' to set it to 0x0, then issue GDB with the 'next' command to step to the next line of code, and this seems to behave as expected. So I put a break point at 'aes_crypt_cbc_wrap', which I expect to be hit, then issue 'continue'. The breakpoint is hit.
Now if I issue either step/next, the same breakpoint is hit. If I put a breakpoint at a function that should be hit, it is not. If I delete the breakpoints and issue 'continue', the same breakpoint is hit.
This is an unexpected behavior. Has anyone seen a correct behavior? Should this be any different if I use the ULINKPLUS probe?
Sine I have not used ULINKPLUS/ULINKPRO, is GDB server/GDB known to work? Does PyOCD work? If not is there an OpenOCD config file anyone can share?
Hi Arun,
In our daily development work, we use ULINK + ARM DS IDE to debug MUSCA S1. And the process is:
1. Add “b .” before SystemInit() in Reset_Handler() in startup_musca.c. 2. Connect debugger in ARM DS IDE. 3. Break the app with debugger. PC will stop at “b .” added in step 1. 4. Add breakpoints you need. 5. Manually set PC to “bl SystemInit” (next instruction of “b .”). 6. Continue to run and the breakpoints will be hit. 7. Press reset button on the board when necessary. 8. Re-connect to the debugger and repeat from step 3.
Officially TF-M does not support GDB neither PyOCD. If you have interests in supporting GDB in TF-M, your contribution is welcome. We are happy to review your patches. : )
Best regards, Xinyu
From: Arun D via TF-M tf-m@lists.trustedfirmware.org Date: Monday, October 23, 2023 at 03:51 To: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Subject: [TF-M] Re: Various issues in successfully using GDB to debug secure apps on Musca S1 connected over USB DAPLink to a Linux host Adding the the last part of my previous message:
When I issue GDB the 'continue' command, nothing is output on the console.
As an additional effort to work around the potential issue, I instrument 'runtime_init_c' (called by 'sprt_main') to have the code spin indefinitely waiting for a volatile unsigned int variable 'gdbSpinHere' varaiable to be changed from the default value of 0xffffffff to 0x0. Using a new debug session, when I issue GDB 'target remote localhost:3333', GDB connects to the PyOCD GDBServer. The backtrace shows 'runtime_init_c' is on the top of the stack. This is as expected.
Next I update 'gdbSpinHere' to set it to 0x0, then issue GDB with the 'next' command to step to the next line of code, and this seems to behave as expected. So I put a break point at 'aes_crypt_cbc_wrap', which I expect to be hit, then issue 'continue'. The breakpoint is hit.
Now if I issue either step/next, the same breakpoint is hit. If I put a breakpoint at a function that should be hit, it is not. If I delete the breakpoints and issue 'continue', the same breakpoint is hit.
This is an unexpected behavior. Has anyone seen a correct behavior? Should this be any different if I use the ULINKPLUS probe?
Sine I have not used ULINKPLUS/ULINKPRO, is GDB server/GDB known to work? Does PyOCD work? If not is there an OpenOCD config file anyone can share? -- TF-M mailing list -- tf-m@lists.trustedfirmware.org To unsubscribe send an email to tf-m-leave@lists.trustedfirmware.org
Hi Xinyiu,
I just want to be sure: do the debugging instructions you mention for using ULINKPLUS + ARM DS IDE work for specifically debugging TF-M (secure+non secure) applications. Like I mention in my message, I do not see any issue so long as the application is completely secure.
PyOCD: I have not contributed anything to it. PyOCD seems to recognize Musca S1 readily. Like Raef Coles mentions in his response, he has used PyOCD to debug Musca S1.
Regards, -Arun.
Hi Arun,
Yes, it works for debugging secure + non secure applications.
Best regards, Xinyu
From: Arun D via TF-M tf-m@lists.trustedfirmware.org Date: Monday, October 23, 2023 at 18:08 To: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Subject: [TF-M] Re: Various issues in successfully using GDB to debug secure apps on Musca S1 connected over USB DAPLink to a Linux host Hi Xinyiu,
I just want to be sure: do the debugging instructions you mention for using ULINKPLUS + ARM DS IDE work for specifically debugging TF-M (secure+non secure) applications. Like I mention in my message, I do not see any issue so long as the application is completely secure.
PyOCD: I have not contributed anything to it. PyOCD seems to recognize Musca S1 readily. Like Raef Coles mentions in his response, he has used PyOCD to debug Musca S1.
Regards, -Arun. -- TF-M mailing list -- tf-m@lists.trustedfirmware.org To unsubscribe send an email to tf-m-leave@lists.trustedfirmware.org
Hey Arun
I also use the pyocd-gdb combination to debug musca-s1, and have wondered before why monitor resets would fail to work (in some cases).
I don't think this is specific to your board, in fact after reading your debugging findings I think I've found the culprit - the TF-M runtime initialization configures resets to only be issuable by the secure world. This makes a lot of sense actually, since it means that during the bootloader stage resets would work, which is why was seeing that happen occasionally.
I'm not sure what needs to be changed in pyocd to issue resets with this config enabled, but for debugging purposes commenting out the reset configuration (musca_s1/target_cfg.c:212) seems to do the trick.
As for the other issues, I've had similar. Stepping in particular I've found to be troublesome, but I've not got any idea how to improve this unfortunately.
Raef
________________________________________ From: Arun D via TF-M tf-m@lists.trustedfirmware.org Sent: 22 October 2023 20:29 To: tf-m@lists.trustedfirmware.org Subject: [TF-M] Various issues in successfully using GDB to debug secure apps on Musca S1 connected over USB DAPLink to a Linux host
Now sure where exactly this belongs, but I already posted question abut the Musca S1 earlier, and this seems relevant. ARM support essentially seem to tell me that they cannot support GDB and they are not certain of DAPLink, and recommend ULINKPLUS debugger with ARM IDE. So wanted to get a sense from anyone who has attempted to use this.
In the following I describe the steps to obtain a base line, then provide description in what way GDB / DAPLink seems to misbehave.
To get a base line: ~~~~~~~~~~~~~ - I build 'samples/hello_world' app using Zephyr 'v3.4.0'. Copy the 'build/zephyr/zephyr.hex' file to '/media/user/MUSCA_S1' directory when the board is USB. Connect a terminal emulator to the UART exposed by USB DAPLink, e.g. 'picocom -b 115200 -y n /dev/ttyACM0'. I can see on the console the followinbg *** Booting Zephyr OS build zephyr-v3.4.0 *** Hello World! musca_s1 If I press the reset button, the app restarts as well. This is as expected. I note that this is entirely running as in secure mode, that it does not use TF-M.
- Next I start 'pyocd-gdbserver'. Then start a GDB session using the 'zephyr.elf' file, 'target remote localhost:3333' to connect it to the gdbserver. It connects ok. So I put a breakpoint at "main", do a 'monitor reset halt', and then continue. The breakpoint at 'main' is hit, I can obtain the backtrace to see it is correct as expected.
So my setup seems good.
Next I attempt to test a non-secure app with TF-M. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Build the app using Zephyr build env 'west build -b v2m_musca_s1_ns samples/tfm_integration/tfm_psa_test/ -DCONFIG_TFM_PSA_TEST_CRYPTO=y'. Copy 'build/zephyr/tfm_merged.hex' to '/media/user/MUSCA_S1'. Connect a terminal emulator, picocom, to the console, and hit the PBON (power on) button. I see the following output as expected on the console:
Musca-S1 Dual Firmware Version 1.9 [WRN] This device was provisioned with dummy keys. This device is NOT SECURE [Sec Thread] Secure image initializing! Booting TF-M 79a6115d3 [INF][Crypto] Initialising HW accelerator... complete. *** Booting Zephyr OS build zephyr-v3.4.0 *** ***** PSA Architecture Test Suite - Version 1.4 ***** Running.. Crypto Suite ****************************************** TEST: 201 | DESCRIPTION: Testing crypto key management APIs | UT: psa_crypto_init [Info] Executing tests from non-secure
This is as expected.
However, if I press the reset button the board does not appear to reset - this is unexpected. Has anyone seen a different behavior? Is the specific board that I have is misbehaving?
- Now I start PyOCD GDB server, start GDB and successfully connect it to the GDB server. The target is halted where it was executing, e.g. Remote debugging using localhost:3333 CC_HalWaitInterruptRND (data=data@entry=1024) at /home/zephyr/zephyrproject/modules/tee/tf-m/trusted-firmware-m/lib/ext/cryptocell-312-runtime/host/src/hal/cc3x/cc_hal.c:107 107 irr = CC_HAL_READ_REGISTER(CC_REG_OFFSET(HOST_RGF, HOST_IRR)); (gdb)
This is as expected.
- Next I put a breakpoint at 'sprt_main', then issue a 'monitor reset halt', then issue a 'continue',
The breakpoint is is NOT hit. I see a blue LED blink at perhaps 4 times per second.
This is unexpected. Has anyone seen this behave correctly? Do I have a board that is misbehaving? -- TF-M mailing list -- tf-m@lists.trustedfirmware.org To unsubscribe send an email to tf-m-leave@lists.trustedfirmware.org
Hey Raef,
Thanks for sharing your experience and observations!
You suggested a possibility that the secure / TF-M handler does not reset the non-secure side when there is a reset detected. That seems reasonable when PyOCD/GDB are used, i.e. when entirely with SW. Is there a TF-M configuration setting that needs to be enabled for this to happen? I am not sure if I remember this 100% for sure of reading this in some TF-M/Zephyr docs. Might anyone else have a suggestion? I will try your suggestion "reset configuration (musca_s1/target_cfg.c:212) seems to do the trick".
So this would mean that even with ARM's ULINK DS IDE, the same issue should be there. Would Xingyu Zhang be able to confirm this?
However, in my first message I mentioned that the pressing the reset button does not seem to reset the Musca S1. I would have expected this to issue a reset to both CPUs. This seems to be a HW issue.
About single stepping/breakpoint continue issues, I wonder if this has something to do with the ITM/ETM implementation.
Regards, -Arun.
Ah apologies, I think I missed that you were having issues with the reset button. I've never seen an issue like that, the reset button seems to work at all time, so I think it might be an issue with your board. The only thing I can think of is that for some reason the reset button will sometimes cause the BL2 image loading to fail, and if you have the BL2 logging turned off there might not be any UART output between the reset and the BL2 failure, so it might look like a reset hadn't happened.
WRT what I was saying earlier, it's not that the non-secure side is being reset and the secure side isn't, it's about the permissions to write the reset register, and hence trigger the reset. So what's happening, I think, is that pyocd is attempting to write the reset register and that write is being ignored because it doesn't originate from secure state. pyocd _can_ issue secure reads and writes (because otherwise it wouldn't be able to read TF-M memory), so I think it should be possible to fix in pyocd.
Also, as you're debugging the CryptoCell driver, you might be interested to note that by default it is compiled in `relwithdebinfo` mode (to save space). You can set `-DMBEDCRYPTO_BUILD_TYPE=debug` to override this, which might aid your debugging experience.
Raef
________________________________________ From: Arun D via TF-M tf-m@lists.trustedfirmware.org Sent: 23 October 2023 11:22 To: tf-m@lists.trustedfirmware.org Subject: [TF-M] Re: Various issues in successfully using GDB to debug secure apps on Musca S1 connected over USB DAPLink to a Linux host
Hey Raef,
Thanks for sharing your experience and observations!
You suggested a possibility that the secure / TF-M handler does not reset the non-secure side when there is a reset detected. That seems reasonable when PyOCD/GDB are used, i.e. when entirely with SW. Is there a TF-M configuration setting that needs to be enabled for this to happen? I am not sure if I remember this 100% for sure of reading this in some TF-M/Zephyr docs. Might anyone else have a suggestion? I will try your suggestion "reset configuration (musca_s1/target_cfg.c:212) seems to do the trick".
So this would mean that even with ARM's ULINK DS IDE, the same issue should be there. Would Xingyu Zhang be able to confirm this?
However, in my first message I mentioned that the pressing the reset button does not seem to reset the Musca S1. I would have expected this to issue a reset to both CPUs. This seems to be a HW issue.
About single stepping/breakpoint continue issues, I wonder if this has something to do with the ITM/ETM implementation.
Regards, -Arun. -- TF-M mailing list -- tf-m@lists.trustedfirmware.org To unsubscribe send an email to tf-m-leave@lists.trustedfirmware.org
Just to close the loop on the issues reported below:
* for the reset issue we discussed with PyOCD maintainers and I believe there will be shortly a fix for that in PyOCD itself that will be integrated in an upcoming new release (i.e. PyOCD issuing secure writes to reset) * for the breakpoints issue, we have been pointed to this patch: Add mps2 an521, reset on loading to Ram and soft-bkpt-as-hard (#1638) · pyocd/pyOCD@6dc261a (github.com)https://github.com/pyocd/pyOCD/commit/6dc261a808b5a90877fdc1378010d37ee30d99c6 . In particular, soft-bkpt-as-hard will force breakpoints to always be HW breakpoints; hopefully this would remove any inconsistent behaviour that might be due to SW breakpoints.
Thanks, Antonio [https://opengraph.githubassets.com/3ef3a11f2752cd0543ac2e27e7f0be556af269a39...]https://github.com/pyocd/pyOCD/commit/6dc261a808b5a90877fdc1378010d37ee30d99c6 Add mps2 an521, reset on loading to Ram and soft-bkpt-as-hard (#1638) · pyocd/pyOCD@6dc261ahttps://github.com/pyocd/pyOCD/commit/6dc261a808b5a90877fdc1378010d37ee30d99c6 * Add support for mps2_an521 * Generate reset when loading to Ram * Add support for soft-bkpt-as-hard github.com
*
________________________________ From: Arun D via TF-M tf-m@lists.trustedfirmware.org Sent: Monday, October 23, 2023 11:22 To: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Subject: [TF-M] Re: Various issues in successfully using GDB to debug secure apps on Musca S1 connected over USB DAPLink to a Linux host
Hey Raef,
Thanks for sharing your experience and observations!
You suggested a possibility that the secure / TF-M handler does not reset the non-secure side when there is a reset detected. That seems reasonable when PyOCD/GDB are used, i.e. when entirely with SW. Is there a TF-M configuration setting that needs to be enabled for this to happen? I am not sure if I remember this 100% for sure of reading this in some TF-M/Zephyr docs. Might anyone else have a suggestion? I will try your suggestion "reset configuration (musca_s1/target_cfg.c:212) seems to do the trick".
So this would mean that even with ARM's ULINK DS IDE, the same issue should be there. Would Xingyu Zhang be able to confirm this?
However, in my first message I mentioned that the pressing the reset button does not seem to reset the Musca S1. I would have expected this to issue a reset to both CPUs. This seems to be a HW issue.
About single stepping/breakpoint continue issues, I wonder if this has something to do with the ITM/ETM implementation.
Regards, -Arun. -- TF-M mailing list -- tf-m@lists.trustedfirmware.org To unsubscribe send an email to tf-m-leave@lists.trustedfirmware.org
Hello Antonio,
The PyOCD patches you refer to are for the MPS2 AN521. Are this also applicable in some ways to the Musca S1? Is there a discussion forum for PyOCD we can follow up on?
Regards, -Arun.
Hi Arun,
they should be three separate work items merged within the same patch:
* Add support for mps2_an521
* Generate reset when loading to Ram
* Add support for soft-bkpt-as-hard
The part that matters here is the option "soft-kpt-as-hard" which should be now a generic option (not specific to mps2_an521 target). Hope this helps.
Thanks, Antonio
________________________________ From: Arun D via TF-M tf-m@lists.trustedfirmware.org Sent: Friday, November 10, 2023 12:29 To: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Subject: [TF-M] Re: Various issues in successfully using GDB to debug secure apps on Musca S1 connected over USB DAPLink to a Linux host
Hello Antonio,
The PyOCD patches you refer to are for the MPS2 AN521. Are this also applicable in some ways to the Musca S1? Is there a discussion forum for PyOCD we can follow up on?
Regards, -Arun. -- TF-M mailing list -- tf-m@lists.trustedfirmware.org To unsubscribe send an email to tf-m-leave@lists.trustedfirmware.org
tf-m@lists.trustedfirmware.org
-
Antonio De Angelis -
Arun D -
Raef Coles -
Xinyu Zhang