Hi,
I was looking at imx_hab_handler, which forwards calls into the BootROM on i.MX SoCs in response to secure monitor calls. The BootROM call (and thus the SiP call) accepts two pointers as arguments, where it writes data.
The plat/imx/imx8m/imx8m*/imx8m*_bl31_setup.c files map the RAM as MT_MEMORY | MT_RW, which would mean that not only is it writable from TF-A side, but it's also mapped cacheable.
I see no explicit cache maintenance in the i.MX SiP code for these two pointer arguments and haven't been successful to find where in TF-A core code if at all, complete D-Cache is flushed on SMC exit. Therefore I ask:
- Don't shared non-secure memory buffers between EL3 and lower EL require explicit cache maintenance?
- Did I miss the place where this cache maintenance is done?
- How did it work so far? barebox does explicit flushing before HAB SiP, invalidation after. U-Boot doesn't do cache maintenance. Both apparently work...
Thanks, Ahmad
--- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
Subject: Cache Maintenance For SiP buffers
Hi,
I was looking at imx_hab_handler, which forwards calls into the BootROM on i.MX SoCs in response to secure monitor calls. The BootROM call (and thus the SiP call) accepts two pointers as arguments, where it writes data.
The plat/imx/imx8m/imx8m*/imx8m*_bl31_setup.c files map the RAM as MT_MEMORY | MT_RW, which would mean that not only is it writable from TF-A side, but it's also mapped cacheable.
I see no explicit cache maintenance in the i.MX SiP code for these two pointer arguments and haven't been successful to find where in TF-A core code if at all, complete D-Cache is flushed on SMC exit. Therefore I ask:
- Don't shared non-secure memory buffers between EL3 and lower EL
require explicit cache maintenance?
Did I miss the place where this cache maintenance is done?
How did it work so far? barebox does explicit flushing before HAB SiP, invalidation after. U-Boot doesn't do cache maintenance. Both apparently work...
You can refer to the docs/plat/imx8m.rst HAB section. ^_^. RAM is the OCRAM, it is not used in the HAB SiP call for write data buffer.
BR
Thanks, Ahmad
Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.p/ engutronix.de%2F&data=05%7C01%7Cping.bai%40nxp.com%7Cea7764cf9a6 740455e8908dbaef6c63e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7 C0%7C638296147136416454%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4 wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% 7C%7C%7C&sdata=eod8jkwJF0ro%2F3F4IRYG2EXpNitykisnag1IwQMJtv8%3D &reserved=0 | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
Hello Jacky,
thanks for your prompt reply.
On 07.09.23 04:03, Jacky Bai wrote:
Subject: Cache Maintenance For SiP buffers
Hi,
I was looking at imx_hab_handler, which forwards calls into the BootROM on i.MX SoCs in response to secure monitor calls. The BootROM call (and thus the SiP call) accepts two pointers as arguments, where it writes data.
The plat/imx/imx8m/imx8m*/imx8m*_bl31_setup.c files map the RAM as MT_MEMORY | MT_RW, which would mean that not only is it writable from TF-A side, but it's also mapped cacheable.
I see no explicit cache maintenance in the i.MX SiP code for these two pointer arguments and haven't been successful to find where in TF-A core code if at all, complete D-Cache is flushed on SMC exit. Therefore I ask:
- Don't shared non-secure memory buffers between EL3 and lower EL
require explicit cache maintenance?
Did I miss the place where this cache maintenance is done?
How did it work so far? barebox does explicit flushing before HAB SiP, invalidation after. U-Boot doesn't do cache maintenance. Both apparently work...
You can refer to the docs/plat/imx8m.rst HAB section. ^_^.
I had checked the docs and while there is a note about why RAM is mapped MT-RW, there is no info about how cache coherency is maintained, hence my question.
RAM is the OCRAM, it is not used in the HAB SiP call for write data buffer.
TF-A RAM is OCRAM, but I care about the write data buffers used by BL33. Those are in DRAM, which is mapped cachable both in secure and non-secure world and may require explicit cache maintenance.
Thanks, Ahmad
BR
Thanks, Ahmad
Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.p/ engutronix.de%2F&data=05%7C01%7Cping.bai%40nxp.com%7Cea7764cf9a6 740455e8908dbaef6c63e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7 C0%7C638296147136416454%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4 wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% 7C%7C%7C&sdata=eod8jkwJF0ro%2F3F4IRYG2EXpNitykisnag1IwQMJtv8%3D &reserved=0 | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
Subject: Re: Cache Maintenance For SiP buffers
Hello Jacky,
thanks for your prompt reply.
On 07.09.23 04:03, Jacky Bai wrote:
Subject: Cache Maintenance For SiP buffers
Hi,
I was looking at imx_hab_handler, which forwards calls into the BootROM on i.MX SoCs in response to secure monitor calls. The BootROM call (and thus the SiP call) accepts two pointers as arguments, where it
writes data.
The plat/imx/imx8m/imx8m*/imx8m*_bl31_setup.c files map the RAM as MT_MEMORY | MT_RW, which would mean that not only is it writable
from
TF-A side, but it's also mapped cacheable.
I see no explicit cache maintenance in the i.MX SiP code for these two pointer arguments and haven't been successful to find where in TF-A core code if at all, complete D-Cache is flushed on SMC exit.
Therefore I ask:
- Don't shared non-secure memory buffers between EL3 and lower EL
require explicit cache maintenance?
Did I miss the place where this cache maintenance is done?
How did it work so far? barebox does explicit flushing before HAB SiP, invalidation after. U-Boot doesn't do cache maintenance. Both apparently work...
You can refer to the docs/plat/imx8m.rst HAB section. ^_^.
I had checked the docs and while there is a note about why RAM is mapped MT-RW, there is no info about how cache coherency is maintained, hence my question.
RAM is the OCRAM, it is not used in the HAB SiP call for write data buffer.
TF-A RAM is OCRAM, but I care about the write data buffers used by BL33. Those are in DRAM, which is mapped cachable both in secure and non-secure world and may require explicit cache maintenance.
The DRAM is mapped as MT_NS in secure world. Should no cache coherence issue.
BR
Thanks, Ahmad
BR
Thanks, Ahmad
Pengutronix e.K. | |
Hello Jacky,
On 07.09.23 07:30, Jacky Bai wrote:
On 07.09.23 04:03, Jacky Bai wrote:
Subject: Cache Maintenance For SiP buffers
- How did it work so far? barebox does explicit flushing before HAB SiP, invalidation after. U-Boot doesn't do cache maintenance. Both apparently work...
You can refer to the docs/plat/imx8m.rst HAB section. ^_^.
I had checked the docs and while there is a note about why RAM is mapped MT-RW, there is no info about how cache coherency is maintained, hence my question.
RAM is the OCRAM, it is not used in the HAB SiP call for write data buffer.
TF-A RAM is OCRAM, but I care about the write data buffers used by BL33. Those are in DRAM, which is mapped cachable both in secure and non-secure world and may require explicit cache maintenance.
The DRAM is mapped as MT_NS in secure world. Should no cache coherence issue.
AFAICS, TF-A has a cacheable mapping of DRAM. If there's no explicit flushing in TF-A, that means that:
- There may be dirty lines in cache on exit to non-secure world - barebox invalidates cache on return from SMC - dirty lines are discarded, RAM is fetched, result is lost
The HAB code is working though, which makes me suspicious, whether I am not missing something.
Cheers, Ahmad
BR
Thanks, Ahmad
BR
Thanks, Ahmad
Pengutronix e.K. | |
Hello Ahmad,
-----Original Message----- From: Ahmad Fatoum a.fatoum@pengutronix.de Sent: Thursday, September 7, 2023 7:56 AM To: Jacky Bai ping.bai@nxp.com; tf-a@lists.trustedfirmware.org; Ye Li ye.li@nxp.com; ZHIZHIKIN Andrey andrey.zhizhikin@leica-geosystems.com Cc: Pengutronix Kernel Team kernel@pengutronix.de Subject: Re: Cache Maintenance For SiP buffers
Hello Jacky,
On 07.09.23 07:30, Jacky Bai wrote:
On 07.09.23 04:03, Jacky Bai wrote:
Subject: Cache Maintenance For SiP buffers
- How did it work so far? barebox does explicit flushing before HAB SiP, invalidation after. U-Boot doesn't do cache maintenance. Both apparently work...
You can refer to the docs/plat/imx8m.rst HAB section. ^_^.
I had checked the docs and while there is a note about why RAM is mapped MT-RW, there is no info about how cache coherency is maintained, hence my question.
RAM is the OCRAM, it is not used in the HAB SiP call for write data buffer.
TF-A RAM is OCRAM, but I care about the write data buffers used by BL33. Those are in DRAM, which is mapped cachable both in secure and non-secure world and may require explicit cache maintenance.
The DRAM is mapped as MT_NS in secure world. Should no cache coherence issue.
AFAICS, TF-A has a cacheable mapping of DRAM. If there's no explicit flushing in TF-A, that means that:
- There may be dirty lines in cache on exit to non-secure world
- barebox invalidates cache on return from SMC
- dirty lines are discarded, RAM is fetched, result is lost
The HAB code is working though, which makes me suspicious, whether I am not missing something.
From what I know, and If I read Section 4 of AN12263 [1], it is the BootROM itself that performs cache maintenance operations. The readout of whether cache is enabled has been missing on certain i.MX6 SOCs (listed in the same section), hence there was a need to inform the BootROM of those SOCs that cache is enabled. This is implemented in U-Boot commit 53c8a510e7 ("arm: imx: hab: Optimise flow of authenticate_image on hab_entry fail"), however the commit description does not explicitly states that this indication is addressed.
Please note, that this applies mainly to HABv4 RVT authenticate_image() API that is primarily used to extend the RoT after the bootloader is fully operational. NXP documentation does not detail if the same applies to BootROM authenticating initial image (PBL or SPL), but I would assume that it does.
Regards, Andrey
Cheers, Ahmad
BR
Thanks, Ahmad
BR
Thanks, Ahmad
Pengutronix e.K. | |
--
Hello Andrey,
On 07.09.23 13:04, ZHIZHIKIN Andrey wrote:
Hello Ahmad,
-----Original Message----- From: Ahmad Fatoum a.fatoum@pengutronix.de Sent: Thursday, September 7, 2023 7:56 AM To: Jacky Bai ping.bai@nxp.com; tf-a@lists.trustedfirmware.org; Ye Li ye.li@nxp.com; ZHIZHIKIN Andrey andrey.zhizhikin@leica-geosystems.com Cc: Pengutronix Kernel Team kernel@pengutronix.de Subject: Re: Cache Maintenance For SiP buffers
Hello Jacky,
On 07.09.23 07:30, Jacky Bai wrote:
On 07.09.23 04:03, Jacky Bai wrote:
Subject: Cache Maintenance For SiP buffers
- How did it work so far? barebox does explicit flushing before HAB SiP, invalidation after. U-Boot doesn't do cache maintenance. Both apparently work...
You can refer to the docs/plat/imx8m.rst HAB section. ^_^.
I had checked the docs and while there is a note about why RAM is mapped MT-RW, there is no info about how cache coherency is maintained, hence my question.
RAM is the OCRAM, it is not used in the HAB SiP call for write data buffer.
TF-A RAM is OCRAM, but I care about the write data buffers used by BL33. Those are in DRAM, which is mapped cachable both in secure and non-secure world and may require explicit cache maintenance.
The DRAM is mapped as MT_NS in secure world. Should no cache coherence issue.
AFAICS, TF-A has a cacheable mapping of DRAM. If there's no explicit flushing in TF-A, that means that:
- There may be dirty lines in cache on exit to non-secure world
- barebox invalidates cache on return from SMC
- dirty lines are discarded, RAM is fetched, result is lost
The HAB code is working though, which makes me suspicious, whether I am not missing something.
From what I know, and If I read Section 4 of AN12263 [1], it is the BootROM itself that performs cache maintenance operations. The readout of whether cache is enabled has been missing on certain i.MX6 SOCs (listed in the same section), hence there was a need to inform the BootROM of those SOCs that cache is enabled. This is implemented in U-Boot commit 53c8a510e7 ("arm: imx: hab: Optimise flow of authenticate_image on hab_entry fail"), however the commit description does not explicitly states that this indication is addressed.
Please note, that this applies mainly to HABv4 RVT authenticate_image() API that is primarily used to extend the RoT after the bootloader is fully operational. NXP documentation does not detail if the same applies to BootROM authenticating initial image (PBL or SPL), but I would assume that it does.
Thanks! That would indeed make sense. That way barebox wouldn't discard dirty lines and U-Boot works, because it happens to have the same memory attributes as TF-A avoiding coherency loss.
A confirmation from NXP that BootROM does cache maintenance for report_status would be appreciated. Jacky?
Thanks, Ahmad
Regards, Andrey
Cheers, Ahmad
BR
Thanks, Ahmad
BR
Thanks, Ahmad
Pengutronix e.K. | |
--
Hello Jacky, Hello Ye,
On 07.09.23 15:49, Ahmad Fatoum wrote:
On 07.09.23 13:04, ZHIZHIKIN Andrey wrote:
From what I know, and If I read Section 4 of AN12263 [1], it is the BootROM itself that performs cache maintenance operations. The readout of whether cache is enabled has been missing on certain i.MX6 SOCs (listed in the same section), hence there was a need to inform the BootROM of those SOCs that cache is enabled. This is implemented in U-Boot commit 53c8a510e7 ("arm: imx: hab: Optimise flow of authenticate_image on hab_entry fail"), however the commit description does not explicitly states that this indication is addressed.
[snip]
Thanks! That would indeed make sense. That way barebox wouldn't discard dirty lines and U-Boot works, because it happens to have the same memory attributes as TF-A avoiding coherency loss.
A confirmation from NXP that BootROM does cache maintenance for report_status would be appreciated. Jacky?
Could you confirm that the BootROM on i.MX8M does cache maintenance internally as on i.MX6?
Thanks, Ahmad
Subject: Re: Cache Maintenance For SiP buffers
Hello Jacky, Hello Ye,
On 07.09.23 15:49, Ahmad Fatoum wrote:
On 07.09.23 13:04, ZHIZHIKIN Andrey wrote:
From what I know, and If I read Section 4 of AN12263 [1], it is the BootROM itself that performs cache maintenance operations. The readout of whether cache is enabled has been missing on certain i.MX6 SOCs (listed in the same section), hence there was a need to inform the BootROM of those SOCs that cache is enabled. This is implemented in
U-Boot commit 53c8a510e7 ("arm: imx: hab:
Optimise flow of authenticate_image on hab_entry fail"), however the commit description does not explicitly states that this indication is
addressed.
[snip]
Thanks! That would indeed make sense. That way barebox wouldn't discard dirty lines and U-Boot works, because it happens to have the same memory attributes as TF-A avoiding coherency loss.
A confirmation from NXP that BootROM does cache maintenance for report_status would be appreciated. Jacky?
Could you confirm that the BootROM on i.MX8M does cache maintenance internally as on i.MX6?
Sorry, it seems I missed your previous mail. ^_^. From what I know, no need to do cache maintenance. HAB will use cache to fast the authentication, but no cacheable data will be write out to be used by the caller. Do you see any data will be used by the caller?
BR
Thanks, Ahmad
-- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.p/ engutronix.de%2F&data=05%7C01%7Cping.bai%40nxp.com%7C2a98c5e5c64 14253ac2408dbba89b413%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0% 7C0%7C638308872812307240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000 %7C%7C%7C&sdata=2SqvTmSvRZn3%2FC4Oh0v9hIIiU7oH0eRqT5tu1LP9Dsc %3D&reserved=0 | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
Hello Jacky,
On 21.09.23 12:34, Jacky Bai wrote:
Subject: Re: Cache Maintenance For SiP buffers Could you confirm that the BootROM on i.MX8M does cache maintenance internally as on i.MX6?
Sorry, it seems I missed your previous mail. ^_^.
No problem. Thanks for the prompt response.
From what I know, no need to do cache maintenance. HAB will use cache to fast the authentication, but no cacheable data will be write out to be used by the caller. Do you see any data will be used by the caller?
barebox does cache maintenance around its FSL_SIP_HAB_REPORT_STATUS SMC. If the BootROM didn't do cache maintenance, I'd have expected data loss in barebox, because dirtied cache lines would be discarded and not a lot happens between return from bootrom and return to normal world that would cause cache line writeback.
I don't observe issues though suggesting that the BootROM does cache maintenance, which would be a good thing, because you don't necessarily want to rely on cache attributes between secure and normal world being the same, but it would be good to know this for a fact.
Thanks, Ahmad
Hello Jacky,
On 21.09.23 12:42, Ahmad Fatoum wrote:
Hello Jacky,
On 21.09.23 12:34, Jacky Bai wrote:
Subject: Re: Cache Maintenance For SiP buffers Could you confirm that the BootROM on i.MX8M does cache maintenance internally as on i.MX6?
Sorry, it seems I missed your previous mail. ^_^.
No problem. Thanks for the prompt response.
From what I know, no need to do cache maintenance. HAB will use cache to fast the authentication, but no cacheable data will be write out to be used by the caller. Do you see any data will be used by the caller?
barebox does cache maintenance around its FSL_SIP_HAB_REPORT_STATUS SMC. If the BootROM didn't do cache maintenance, I'd have expected data loss in barebox, because dirtied cache lines would be discarded and not a lot happens between return from bootrom and return to normal world that would cause cache line writeback.
I don't observe issues though suggesting that the BootROM does cache maintenance, which would be a good thing, because you don't necessarily want to rely on cache attributes between secure and normal world being the same, but it would be good to know this for a fact.
Are you able to confirm that the i.MX8M BootROM does in fact do cache maintenance itself for HAB buffers it writes to?
Thanks again, Ahmad
Thanks, Ahmad
Hello Jacky, Hello Ye,
On 21.10.23 18:18, Ahmad Fatoum wrote:
Hello Jacky,
On 21.09.23 12:42, Ahmad Fatoum wrote:
Hello Jacky,
On 21.09.23 12:34, Jacky Bai wrote:
Subject: Re: Cache Maintenance For SiP buffers Could you confirm that the BootROM on i.MX8M does cache maintenance internally as on i.MX6?
Sorry, it seems I missed your previous mail. ^_^.
No problem. Thanks for the prompt response.
From what I know, no need to do cache maintenance. HAB will use cache to fast the authentication, but no cacheable data will be write out to be used by the caller. Do you see any data will be used by the caller?
barebox does cache maintenance around its FSL_SIP_HAB_REPORT_STATUS SMC. If the BootROM didn't do cache maintenance, I'd have expected data loss in barebox, because dirtied cache lines would be discarded and not a lot happens between return from bootrom and return to normal world that would cause cache line writeback.
I don't observe issues though suggesting that the BootROM does cache maintenance, which would be a good thing, because you don't necessarily want to rely on cache attributes between secure and normal world being the same, but it would be good to know this for a fact.
Are you able to confirm that the i.MX8M BootROM does in fact do cache maintenance itself for HAB buffers it writes to?
any news on whether BL33 should do cache maintenance on i.MX8M or not (or if it doesn't matter)?
Cheers, Ahmad
Thanks again, Ahmad
Thanks, Ahmad
tf-a@lists.trustedfirmware.org
-
Ahmad Fatoum -
Jacky Bai -
ZHIZHIKIN Andrey