[AMD Official Use Only - AMD Internal Distribution Only]
Hi Mathew,
Thank you for quick response and shared required information.
Regards, Prasad.
From: Rohit Mathew Rohit.Mathew@arm.com Sent: Tuesday, January 6, 2026 9:21 PM To: Kummari, Prasad Prasad.Kummari@amd.com; Sammit Joshi Sammit.Joshi@arm.com; scan-admin--- via TF-A tf-a@lists.trustedfirmware.org Cc: Belsare, Akshay akshay.belsare@amd.com; Bollapalli, Maheedhar Sai MaheedharSai.Bollapalli@amd.com; Simek, Michal michal.simek@amd.com Subject: Re: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi Prasad,
Thanks for the info. We tried booting on few different platforms (ACPI-Linux/DT-Linux/TFTF) from our end with Suspend and LTO enabled and boot seems to go through without any problems - so we still can't reproduce the issue on our end. Could I request the following info from your end to see if we can spot something?
* Do you see this happening only for the said platform (In case you have other platforms)
The issue is currently observed only on our ZynqMP (Cortex-A53) based platform. We do not see the issue on other internal platforms we have access to. The failure is reproducible on ZynqMP Platform, we have enabled LTO.
* Since you had debug logs added, could we check if this happens on the first/specific CPU suspend or is this quite random? Were you able to trace it to a specific function/line within the suspend path where the crash occurs? It appears to be quite random. Added debug logs in psci_cpu_suspend(). Although BL31 NOTICE logs and Linux printk messages are interleaved and somewhat noisy, they are still understandable. I've attached the full logs for reference with DEBUG=1 and DEBUG=0. LOGs DEBUG=1: [ 4.966988] zynqmp-dpsub fd4a0000.display: [drm] Cannot find any crtc or sizes NOTICE: CPU2: psci_cpu_suspend() base=0x12280 next=0x125c0 delta=0x340 NOTICE: CPU3: psci_cpu_suspend() base=0x125c0 next=0x11c00 delta=0xfffffffffffff640 NOTICE: CPU3: psci_cpu_suspend() base=0x125c0 next=0x11c00 delta=0xfffffffffffff640 NOTICE: CPU0: psci_cpu_suspend() base=0x11c00 next=0x11f40 delta=0x340 nUOTnhIaCnEd:l e dCP EUx3c: epptsciio_nc ipun_ sEuLs3p.e ---> unexpectedly entering EL3 =0(3)0 b a se = 0 x 1 2 5 c0= n0exx0t00=000x01010c00000 0de0l00t2a 0xf0f f ff f f f f f f ff 6 4=0 x0000000000000000 x1 = 0x00000000000111f8 NOTICE: CPU1: psci_cpu_suspend() base=0x11f40 next=0x12280 delta=0x340 NOTICE: CPU0: psci_cpu_suspend() base=0x11c00 next=0x11f40 delta=0x340 NOTICE: CPU3: psci_cpu_suspend() base=0x125c0 next=0x11c00 delta=0xfffffffffffff640 NOTICE: CPU1: psci_cpu_suspend() base=0x11f40 next=0x12280 delta=0x340 --> continues prints and hang.
* It was stated that CPUs are unexpectedly entering EL3. Can I check if the entry itself was not expected or if there was a problem further down the line ie in the suspend path following the entry? Exception Register Dump: x30 = 0x0000000000000002 x0 = 0x0000000000000000 x1 = 0x0000000000012538 x2 = 0x00000000000134c0 x3 = 0x000000000000003f x4 = 0x00000000000002c0 x5 = 0x0000000055540000 x6 = 0x000000000000b554 x7 = 0x00000000f9010000 x8 = 0x0000000000000008 x9 = 0x000000000000b1b0 x10 = 0x000000002000ff00 x11 = 0x000000000000b554 x12 = 0x00000000f9010400 x13 = 0x0000000000001ff0 x14 = 0x0000000000000001 x15 = 0x0000000000001100 x16 = 0xaa69440080c94020 x17 = 0x51453496800300f0 x18 = 0xa1c6bb25d660316b x19 = 0x00012801026a8386 x20 = 0x246f2823c0616a0c x21 = 0x2a0495a48c060008 x22 = 0x0020241c01920c03 x23 = 0x271802d906c4c238 x24 = 0xd56233b4061dd10c x25 = 0xa4b115480a80090f x26 = 0x8d098c0181808610 x[ 3.875646] mmc0: new high speed SDHC card at address aaaa 27 = 0x2241009a000000c0 x28 = 0x1c180f2194984856 x29 = 0x301c2a610c800908 scr_el3 = 0x0000000000000238 sctlr_el3 = 0x0000000030cd183f cptr_el3 = 0x0000000000000000 tcr_el3 = 0x0000000080803520 daif = 0x00000000000003c0 mair_el3 = 0x00000000004400ff spsr_el3 = 0x00000000200002cc elr_el3 = 0x0000000000000002 ttbr0_el3 = 0x0000000000012ac0 esr_el3 = 0x000000008a000000 far_el3 = 0x0000000000000002 mpidr_el1 = 0x0000000080000003 sp_el0 = 0x0000000000012540 isr_el1 = 0x0000000000000000 dacr32_el2 = 0x0000000000000000 ifsr32_el2 = 0x0000000000000000 cpuectlr_el1 = 0x0000000000000040 cpumerrsr_el1 = 0x000000001a000040 l2merrsr_el1 = 0x0000000010100008 cpuactlr_el1 = 0x00001000090ca000 gicc_hppir = 0x00000000000003fe gicc_ahppir = 0x0000000000000801 gicc_ctlr = 0x00000000000001e9 gicd_ispendr regs (Offsets 0x200-0x278) Offset Value 0x200: 0x0000000000000012 0x208: 0x0000000000000000 0x210: 0x0000000000000000 0x218: 0x0000000000000000 0x220: 0x0000000000000000 0x228: 0x0000000000000000 0x230: 0x0000000000000000 0x238: 0x0000000000000000 0x240: 0x0000000000000000 0x248: 0x0000000000000000 0x250: 0x0000000000000000 0x258: 0x0000000000000000 0x260: 0x0000000000000000 0x268: 0x0000000000000000 0x270: 0x0000000000000000 0x278: 0x0000000000000000 cci_snoop_ctrl_cluster0x100000000c0000003 cci_snoop_ctrl_cluster1x100000000c0000000
CPU state with debugger:
xsdb% ta 1 PS TAP 2 PMU 3 MicroBlaze PMU (Sleeping. No clock) 4 PL 5 PSU 6 RPU 7 Cortex-R5 #0 (Halted) 8 Cortex-R5 #1 (Lock Step Mode) 9 APU 10* Cortex-A53 #0 (External debug access is disabled) 11 Cortex-A53 #1 (No Power) 12 Cortex-A53 #2 (Running) 13 Cortex-A53 #3 (Running) xsdb% ta 1 PS TAP 2 PMU 3 MicroBlaze PMU (Sleeping. No clock) 4 PL 5 PSU 6 RPU 7 Cortex-R5 #0 (Halted) 8 Cortex-R5 #1 (Lock Step Mode) 9 APU 10* Cortex-A53 #0 (APB AP transaction error, DAP status 0x30000021) 11 Cortex-A53 #1 (Running) 12 Cortex-A53 #2 (Running) 13 Cortex-A53 #3 (No Power) xsdb%
* Could you share the exact commit you are on for TF-A Merge "docs(changelog): changelog for v2.14 release" into integration * ARM-software/arm-trusted-firmware@1d5aa93https://github.com/ARM-software/arm-trusted-firmware/commit/1d5aa939bc8d3d892e2ed9945fa50e36a1a924cc
* Could you share the build configuration for the platform (all build-flags) make -j20 RESET_TO_BL31=1 PLAT=zynqmp bl31 IPI_CRC_CHECK=1 DEBUG=1 In DEBUG mode ZynqMP will run DDR start address: 0x1000, without debug it will run OCM start address 0x00000000fffea000
* Would it be possible to share the BL31 logs (so we can see the crash register logs) and Linux logs (just to correlate) as well as bl31.dump/bl31.map file for the failure case (If you can share for the working case as well, that would be great) Attached are the working and non-working dumps and logs for NUMA_AWARE_PER_CPU with DDR and OCM. In the working case, LTO is disabled. CPU state with debugger with OCM BL31.elf: xsdb% ta 1 PS TAP 2 PMU 3 MicroBlaze PMU (Sleeping. No clock) 4 PL 5 PSU 6 RPU 7 Cortex-R5 #0 (Halted) 8 Cortex-R5 #1 (Lock Step Mode) 9 APU 10* Cortex-A53 #0 (Running) 11 Cortex-A53 #1 (No Power) 12 Cortex-A53 #2 (Power On Reset) 13 Cortex-A53 #3 (Running) xsdb% Info: Cortex-A53 #2 (target 12) Stopped at 0x0 (Cannot resume. APB AP transaction error, DAP status 0x30000021) xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power) xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power) xsdb% Info: Cortex-A53 #2 (target 12) Stopped at 0x0 (Cannot resume. APB AP transaction error, DAP status 0x30000021) xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power) xsdb% Info: Cortex-A53 #0 (target 10) Stopped at 0xfffea194 (Reset Catch) xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power) xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power) xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power) xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power) xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power) xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power) xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power) xsdb% ta 1 PS TAP 2 PMU 3 MicroBlaze PMU (Sleeping. No clock) 4 PL 5 PSU 6 RPU 7 Cortex-R5 #0 (Halted) 8 Cortex-R5 #1 (Lock Step Mode) 9 APU 10* Cortex-A53 #0 (Reset Catch, EL3(S)/A64) 11 Cortex-A53 #1 (No Power) 12 Cortex-A53 #2 (Running) 13 Cortex-A53 #3 (Running) xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power)
Regards, Rohit
From: Kummari, Prasad <Prasad.Kummari@amd.commailto:Prasad.Kummari@amd.com> Date: Tuesday, 6 January 2026 at 06:34 To: Rohit Mathew <Rohit.Mathew@arm.commailto:Rohit.Mathew@arm.com>, Sammit Joshi <Sammit.Joshi@arm.commailto:Sammit.Joshi@arm.com>, scan-admin--- via TF-A <tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org> Cc: Belsare, Akshay <akshay.belsare@amd.commailto:akshay.belsare@amd.com>, Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.commailto:MaheedharSai.Bollapalli@amd.com>, Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: RE: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry
[AMD Official Use Only - AMD Internal Distribution Only]
Hi Mathew,
Yes, if the CPU_IDLE-related configurations listed below are removed while compiling the kernel image, the hang is no longer observed.
Configs: CONFIG_CPU_IDLE=y CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y CONFIG_CPU_IDLE_GOV_MENU=y CONFIG_DT_IDLE_STATES=y
Observed Behavior
* System later triggers an Unhandled Exception in EL3 CPUs enter psci_cpu_suspend(). Debug prints confirm execution reaches the suspend path. * The affected CPUs enter Reset Catch at EL3 and hang at the bl31_warm_entrypoint()->el3_entrypoint_common pc.
Regards, Prasad.
From: Rohit Mathew <Rohit.Mathew@arm.commailto:Rohit.Mathew@arm.com> Sent: Tuesday, January 6, 2026 4:37 AM To: Kummari, Prasad <Prasad.Kummari@amd.commailto:Prasad.Kummari@amd.com>; Sammit Joshi <Sammit.Joshi@arm.commailto:Sammit.Joshi@arm.com>; scan-admin--- via TF-A <tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org> Cc: Belsare, Akshay <akshay.belsare@amd.commailto:akshay.belsare@amd.com>; Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.commailto:MaheedharSai.Bollapalli@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi Prasad,
Thanks for your mail. I'm trying to reproduce this at our end. Could I check if you see the boot progressing to shell without the CPU idle dependencies?
Regards, Rohit
From: Kummari, Prasad <Prasad.Kummari@amd.commailto:Prasad.Kummari@amd.com> Date: Monday, 5 January 2026 at 18:04 To: Sammit Joshi <Sammit.Joshi@arm.commailto:Sammit.Joshi@arm.com>, Rohit Mathew <Rohit.Mathew@arm.commailto:Rohit.Mathew@arm.com>, scan-admin--- via TF-A <tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org> Cc: Belsare, Akshay <akshay.belsare@amd.commailto:akshay.belsare@amd.com>, Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.commailto:MaheedharSai.Bollapalli@amd.com>, Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry
[AMD Official Use Only - AMD Internal Distribution Only]
Hello Maintainers,
We are observing a reproducible runtime regression on the ZynqMP (Cortex-A53) platform after enabling LTO (ENABLED_LTO=1) and merging the changes from the topic NUMA_AWARE_PER_CPU into our integration branch (https://github.com/ARM-software/arm-trusted-firmware/commit/7303319b3823e9e3...).
Summary of the issue
1. Baseline behavior
* Platform: ZynqMP (Cortex-A53)
* Configuration: ENABLED_LTO=1
* Without NUMA_AWARE_PER_CPU: Linux boots and runs stably
* After merging NUMA_AWARE_PER_CPU, Linux boots but hangs during runtime
* During the hang, CPUs are observed to unexpectedly re-enter EL3
* Re-entry into EL3 should not occur during normal Linux runtime execution and strongly suggests corruption or mismanagement of PSCI and/or per-CPU state(arm-trusted-firmware/lib/per_cpu/aarch64/per_cpu_asm.S at master * ARM-software/arm-trusted-firmwarehttps://github.com/ARM-software/arm-trusted-firmware/blob/master/lib/per_cpu/aarch64/per_cpu_asm.S#L28)
* Reverting the NUMA_AWARE_PER_CPU changes restores stable Linux execution
* The issue is reproducible only when NUMA_AWARE_PER_CPU is present
* This clearly identifies NUMA_AWARE_PER_CPU as the regression source
1. Suspect with LTO
* With NUMA_AWARE_PER_CPU enabled, LTO breaks the per-CPU base calculation
* BL31 contains hand-written assembly that relies on linker-script symbols (e.g., per-CPU section boundaries)
* Under LTO, symbol placement and retention are no longer guaranteed in the same way, leading to incorrect per-CPU base computation
* This results in corrupted per-CPU data and subsequent erroneous PSCI suspend behavior (EL3 re-entry)
1. CPU idle dependency
* The following kernel configuration options are enabled:
* CONFIG_CPU_IDLE=y
* CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y
* CONFIG_CPU_IDLE_GOV_MENU=y
* CONFIG_DT_IDLE_STATES=y
* This further suggests the issue is triggered during CPU idle / suspend-resume paths, where correct per-CPU state handling is critical
Based on the above:
* This is specific to NUMA_AWARE_PER_CPU combined with LTO * The failure mode points to per-CPU base calculation and PSCI state corruption * Reverting NUMA_AWARE_PER_CPU fully restores stability on ZynqMP
We wanted to report this issue upstream and seek guidance on:
* Whether NUMA_AWARE_PER_CPU is expected to be LTO-safe on platforms relying on linker-defined per-CPU sections * Or if additional constraints / fixes are required for platforms like ZynqMP
We are happy to provide further logs, configuration details, or help to fixes.
Regards, Prasad Kummari