Hi,
When a core is in debug recovery mode its caches are not invalidated upon reset, so the L1 and L2 cache contents from before reset are observable after reset. Similarly, debug recovery mode of DynamIQ cluster ensures that contents of the shared L3 cache are also not invalidated upon transition to On mode.
A common use case of booting cores in debug recovery mode is to boot with caches disabled and preserve the caches until a point where software can dump the caches and retrieve their contents. TF-A however unconditionally cleans and invalidates caches at multiple points during boot, e.g. in bl31_entrypoint when cleaning bss and .data sections. This will not only lose the cache content needed for debugging but will potentially corrupt memory as well, leading to bugs when booting in recovery mode.
Can we make CMOs in lib/aarch64/cache_helpers.S conditional upon some platform hook to address above scenario? Happy to work on a patch if the idea of conditional CMOs makes sense.
Thanks, Okash
Hi Okash,
From an initial scan of the CPU TRMs, the caches (L1 I-Cache/D-Cache and/or L2 Cache) supported for "Direct access to internal memory" under this feature could be different for different CPUs. So, it is reasonable to have platform hook to do this and take of this for the caches required/applicable per the platform config and convert the "unconditional cleans and invalidates caches" that TF-A does, to something conditional to the "core power mode not equal to Debug Recovery".
We would be happy to help review any patch for this.
--Bipin
Hi Bipin,
Thanks for your feedback. I was thinking of something quite simple. E.g. in each of the assembly routines in lib/aarch64/cache_helpers.S (flush_dcache_range, clean_dcache_range, inv_dcache_range etc), add a call to following macro right at the beginning:
``` .macro check_plat mov x2, x0 bl plat_can_cmo cbnz x0, 1f ret 1: mov x0, x2 .endm ```
Each platform can implement plat_can_cmo. Otherwise a default weak implementation will be chosen, such as
``` func plat_can_cmo mov x0, #1 ret endfunc plat_can_cmo ``` Then it's up to the platform to determine whether they want cache clean or invalidates depending upon their specific conditions, which might well be checking CPU debug/recovery mode. What do you think?
Best regards, Okash
On Mon, Oct 31, 2022 at 5:35 PM Bipin Ravi via TF-A tf-a@lists.trustedfirmware.org wrote:
Hi Okash,
From an initial scan of the CPU TRMs, the caches (L1 I-Cache/D-Cache and/or L2 Cache) supported for "Direct access to internal memory" under this feature could be different for different CPUs. So, it is reasonable to have platform hook to do this and take of this for the caches required/applicable per the platform config and convert the "unconditional cleans and invalidates caches" that TF-A does, to something conditional to the "core power mode not equal to Debug Recovery".
We would be happy to help review any patch for this.
--Bipin
TF-A mailing list -- tf-a@lists.trustedfirmware.org To unsubscribe send an email to tf-a-leave@lists.trustedfirmware.org
Hi Okash,
This looks good to me. This enables platform to decide whether to skip the cache invalidate based on other conditions (if there is memory retention of caches required based on power state as an example) in addition to the original case of "Debug Recovery Mode".
Hi Okash
Your use-case is reasonable, but I'm a bit concerned about imposing the runtime cost of a platform call on all platforms to cater for the subset of platforms that support this mode. Can this be wrapped in a build flag (off by default)?
Also, I wonder if something more generically useful could be done here? E.g. is there some mode detection logic in your plat_can_cmo implementation that can be brought into generic code? I'm not familiar enough with this feature to know if that's possible.
Regards
Dan.
-----Original Message----- From: Okash Khawaja via TF-A tf-a@lists.trustedfirmware.org Sent: 31 October 2022 18:11 To: Bipin Ravi Bipin.Ravi@arm.com Cc: tf-a@lists.trustedfirmware.org Subject: [TF-A] Re: Conditional CMOs
Hi Bipin,
Thanks for your feedback. I was thinking of something quite simple. E.g. in each of the assembly routines in lib/aarch64/cache_helpers.S (flush_dcache_range, clean_dcache_range, inv_dcache_range etc), add a call to following macro right at the beginning:
.macro check_plat mov x2, x0 bl plat_can_cmo cbnz x0, 1f ret 1: mov x0, x2 .endmEach platform can implement plat_can_cmo. Otherwise a default weak implementation will be chosen, such as
func plat_can_cmo mov x0, #1 ret endfunc plat_can_cmoThen it's up to the platform to determine whether they want cache clean or invalidates depending upon their specific conditions, which might well be checking CPU debug/recovery mode. What do you think?
Best regards, Okash
On Mon, Oct 31, 2022 at 5:35 PM Bipin Ravi via TF-A <tf- a@lists.trustedfirmware.org> wrote:
Hi Okash,
From an initial scan of the CPU TRMs, the caches (L1 I-Cache/D-Cache
and/or L2 Cache) supported for "Direct access to internal memory" under this feature could be different for different CPUs. So, it is reasonable to have platform hook to do this and take of this for the caches required/applicable per the platform config and convert the "unconditional cleans and invalidates caches" that TF-A does, to something conditional to the "core power mode not equal to Debug Recovery".
We would be happy to help review any patch for this.
--Bipin
TF-A mailing list -- tf-a@lists.trustedfirmware.org To unsubscribe send an email to tf-a-leave@lists.trustedfirmware.org
-- TF-A mailing list -- tf-a@lists.trustedfirmware.org To unsubscribe send an email to tf-a-leave@lists.trustedfirmware.org
Hi Dan,
Here's the gerrit link: https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/17611. Replies inlined below.
On Mon, Nov 7, 2022 at 12:25 PM Dan Handley Dan.Handley@arm.com wrote:
Hi Okash
Your use-case is reasonable, but I'm a bit concerned about imposing the runtime cost of a platform call on all platforms to cater for the subset of platforms that support this mode. Can this be wrapped in a build flag (off by default)?
Sure, I can do that. The default implementation is quite simple though.
Also, I wonder if something more generically useful could be done here? E.g. is there some mode detection logic in your plat_can_cmo implementation that can be brought into generic code? I'm not familiar enough with this feature to know if that's possible.
I think detecting debug recovery mode is not consistent across different cores. Also it's not just CPUs which could be in debug recovery mode. The DSU logic could also be in that mode. Unless we have a way to test all different configurations, there is a possibility that some config might get missed. With the default implementation the current behaviour will remain unchanged.
Regards
Dan.
-----Original Message----- From: Okash Khawaja via TF-A tf-a@lists.trustedfirmware.org Sent: 31 October 2022 18:11 To: Bipin Ravi Bipin.Ravi@arm.com Cc: tf-a@lists.trustedfirmware.org Subject: [TF-A] Re: Conditional CMOs
Hi Bipin,
Thanks for your feedback. I was thinking of something quite simple. E.g. in each of the assembly routines in lib/aarch64/cache_helpers.S (flush_dcache_range, clean_dcache_range, inv_dcache_range etc), add a call to following macro right at the beginning:
.macro check_plat mov x2, x0 bl plat_can_cmo cbnz x0, 1f ret 1: mov x0, x2 .endmEach platform can implement plat_can_cmo. Otherwise a default weak implementation will be chosen, such as
func plat_can_cmo mov x0, #1 ret endfunc plat_can_cmoThen it's up to the platform to determine whether they want cache clean or invalidates depending upon their specific conditions, which might well be checking CPU debug/recovery mode. What do you think?
Best regards, Okash
On Mon, Oct 31, 2022 at 5:35 PM Bipin Ravi via TF-A <tf- a@lists.trustedfirmware.org> wrote:
Hi Okash,
From an initial scan of the CPU TRMs, the caches (L1 I-Cache/D-Cache
and/or L2 Cache) supported for "Direct access to internal memory" under this feature could be different for different CPUs. So, it is reasonable to have platform hook to do this and take of this for the caches required/applicable per the platform config and convert the "unconditional cleans and invalidates caches" that TF-A does, to something conditional to the "core power mode not equal to Debug Recovery".
We would be happy to help review any patch for this.
--Bipin
TF-A mailing list -- tf-a@lists.trustedfirmware.org To unsubscribe send an email to tf-a-leave@lists.trustedfirmware.org
-- TF-A mailing list -- tf-a@lists.trustedfirmware.org To unsubscribe send an email to tf-a-leave@lists.trustedfirmware.org
tf-a@lists.trustedfirmware.org
-
bipin.ravi＠arm.com -
Dan Handley -
Okash Khawaja