Hi Prasad,

Thanks for sharing all the build artefacts and requested information. The register states (X2 and SP_EL0) + map/dump files for the DDR boot shows that CPU2 went into exception but the exact point of crash is unfortunately not recoverable from the shared artefacts - some registers are lost due to scrambled logs and the other ones don’t show any relatable offsets/addresses which can be traced back from the dump and map files. The OCM boot logs doesn’t show an exception so there isn’t much to decode from those logs. 

The map and dump files for the failed LTO builds doesn’t build look different from the passing ones we have on our end (We have 3 passing builds/3 different platforms). The order of how the objects are placed in the section changes between an LTO and non-LTO build, but per-cpu accessors are object-order-agnostic within a CPUs space in general.

You did mention that the cores that went into crash then further woke up in bl31_warmboot_entrypoint. Could this mean that the CPU suspend sequence from the Zync’s PMU was issued before the core went into an exception and this caused the core to not land at the WFI but instead go into an exception? Is this something that could be checked relatively easily? Also, do you know the exact instruction at which such cores were hung in the warm boot entrypoint? If we don’t get much information, we might have to think about adding a debug patch to retrieve more about the crash. Could we also check:

It seems that not everyone is getting the mails from the mailing list at from Arm side. Would it be okay to move the rest of the discussion/debug to discord TF-A so that everyone can participate? Let us know.

 Regards,
Rohit

From: Kummari, Prasad <Prasad.Kummari@amd.com>
Date: Tuesday, 6 January 2026 at 18:11
To: Rohit Mathew <Rohit.Mathew@arm.com>, Sammit Joshi <Sammit.Joshi@arm.com>, scan-admin--- via TF-A <tf-a@lists.trustedfirmware.org>
Cc: Belsare, Akshay <akshay.belsare@amd.com>, Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.com>, Simek, Michal <michal.simek@amd.com>, Kummari, Prasad <Prasad.Kummari@amd.com>
Subject: RE: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry

[AMD Official Use Only - AMD Internal Distribution Only]


++  @Kummari, Prasad

 

From: Kummari, Prasad
Sent: Tuesday, January 6, 2026 11:31 PM
To: 'Rohit Mathew' <Rohit.Mathew@arm.com>; Sammit Joshi <Sammit.Joshi@arm.com>; scan-admin--- via TF-A <tf-a@lists.trustedfirmware.org>
Cc: Belsare, Akshay <akshay.belsare@amd.com>; Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.com>; Simek, Michal <michal.simek@amd.com>
Subject: RE: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry

 

Hi Mathew,

Thank you for quick response and shared required information.

 

Regards,

Prasad.

 

From: Rohit Mathew <Rohit.Mathew@arm.com>
Sent: Tuesday, January 6, 2026 9:21 PM
To: Kummari, Prasad <Prasad.Kummari@amd.com>; Sammit Joshi <Sammit.Joshi@arm.com>; scan-admin--- via TF-A <tf-a@lists.trustedfirmware.org>
Cc: Belsare, Akshay <akshay.belsare@amd.com>; Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.com>; Simek, Michal <michal.simek@amd.com>
Subject: Re: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry

 

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

 

Hi Prasad,

 

Thanks for the info. We tried booting on few different platforms (ACPI-Linux/DT-Linux/TFTF) from our end with Suspend and LTO enabled and boot seems to go through without any problems - so we still can’t reproduce the issue on our end.  Could I request the following info from your end to see if we can spot something?

 

·       Do you see this happening only for the said platform (In case you have other platforms)

The issue is currently observed only on our ZynqMP (Cortex-A53) based platform. We do not see the issue on other internal platforms we have access to. The failure is reproducible on ZynqMP Platform, we have enabled LTO.

  • Since you had debug logs added, could we check if this happens on the first/specific CPU suspend or is this quite random? Were you able to trace it to a specific function/line within the suspend path where the crash occurs?

              It appears to be quite random. Added debug logs in psci_cpu_suspend(). Although BL31 NOTICE logs and Linux printk messages are interleaved and somewhat noisy,                      

              they are still understandable. I’ve attached the full logs for reference with DEBUG=1 and DEBUG=0.

              LOGs DEBUG=1:

        [    4.966988] zynqmp-dpsub fd4a0000.display: [drm] Cannot find any crtc or sizes

         NOTICE:  CPU2: psci_cpu_suspend() base=0x12280 next=0x125c0 delta=0x340

        NOTICE:  CPU3: psci_cpu_suspend() base=0x125c0 next=0x11c00 delta=0xfffffffffffff640

        NOTICE:  CPU3: psci_cpu_suspend() base=0x125c0 next=0x11c00 delta=0xfffffffffffff640

        NOTICE:  CPU0: psci_cpu_suspend() base=0x11c00 next=0x11f40 delta=0x340

        nUOTnhIaCnEd:l e dCP EUx3c: epptsciio_nc ipun_ sEuLs3p.e  -à unexpectedly entering EL3

         =0(3)0  b a  se = 0 x 1 2 5  c0=  n0exx0t00=000x01010c00000 0de0l00t2a

         0xf0f f  ff f f f f f f  ff 6 4=0

         x0000000000000000

         x1             = 0x00000000000111f8

NOTICE:  CPU1: psci_cpu_suspend() base=0x11f40 next=0x12280 delta=0x340

NOTICE:  CPU0: psci_cpu_suspend() base=0x11c00 next=0x11f40 delta=0x340

NOTICE:  CPU3: psci_cpu_suspend() base=0x125c0 next=0x11c00 delta=0xfffffffffffff640

             NOTICE:  CPU1: psci_cpu_suspend() base=0x11f40 next=0x12280 delta=0x340  à continues prints and hang.

  • It was stated that CPUs are unexpectedly entering EL3. Can I check if the entry itself was not expected or if there was a problem further down the line ie in the suspend path following the entry?

                                          Exception Register Dump:

x30            = 0x0000000000000002

x0             = 0x0000000000000000

x1             = 0x0000000000012538

x2             = 0x00000000000134c0

x3             = 0x000000000000003f

x4             = 0x00000000000002c0

x5             = 0x0000000055540000

x6             = 0x000000000000b554

x7             = 0x00000000f9010000

x8             = 0x0000000000000008

x9             = 0x000000000000b1b0

x10            = 0x000000002000ff00

x11            = 0x000000000000b554

x12            = 0x00000000f9010400

x13            = 0x0000000000001ff0

x14            = 0x0000000000000001

x15            = 0x0000000000001100

x16            = 0xaa69440080c94020

x17            = 0x51453496800300f0

x18            = 0xa1c6bb25d660316b

x19            = 0x00012801026a8386

x20            = 0x246f2823c0616a0c

x21            = 0x2a0495a48c060008

x22            = 0x0020241c01920c03

x23            = 0x271802d906c4c238

x24            = 0xd56233b4061dd10c

x25            = 0xa4b115480a80090f

x26            = 0x8d098c0181808610

x[    3.875646] mmc0: new high speed SDHC card at address aaaa

27            = 0x2241009a000000c0

x28            = 0x1c180f2194984856

x29            = 0x301c2a610c800908

scr_el3        = 0x0000000000000238

sctlr_el3      = 0x0000000030cd183f

cptr_el3       = 0x0000000000000000

tcr_el3        = 0x0000000080803520

daif           = 0x00000000000003c0

mair_el3       = 0x00000000004400ff

spsr_el3       = 0x00000000200002cc

elr_el3        = 0x0000000000000002

ttbr0_el3      = 0x0000000000012ac0

esr_el3        = 0x000000008a000000

far_el3        = 0x0000000000000002

mpidr_el1      = 0x0000000080000003

sp_el0         = 0x0000000000012540

isr_el1        = 0x0000000000000000

dacr32_el2     = 0x0000000000000000

ifsr32_el2     = 0x0000000000000000

cpuectlr_el1   = 0x0000000000000040

cpumerrsr_el1  = 0x000000001a000040

l2merrsr_el1   = 0x0000000010100008

cpuactlr_el1   = 0x00001000090ca000

gicc_hppir     = 0x00000000000003fe

gicc_ahppir    = 0x0000000000000801

gicc_ctlr      = 0x00000000000001e9

gicd_ispendr regs (Offsets 0x200-0x278)

Offset                              Value

0x200:               0x0000000000000012

0x208:               0x0000000000000000

0x210:               0x0000000000000000

0x218:               0x0000000000000000

0x220:               0x0000000000000000

0x228:               0x0000000000000000

0x230:               0x0000000000000000

0x238:               0x0000000000000000

0x240:               0x0000000000000000

0x248:               0x0000000000000000

0x250:               0x0000000000000000

0x258:               0x0000000000000000

0x260:               0x0000000000000000

0x268:               0x0000000000000000

0x270:               0x0000000000000000

0x278:               0x0000000000000000

cci_snoop_ctrl_cluster0x100000000c0000003

cci_snoop_ctrl_cluster1x100000000c0000000

 

CPU state with debugger:

xsdb% ta

  1  PS TAP

     2  PMU

        3  MicroBlaze PMU (Sleeping. No clock)

     4  PL

  5  PSU

     6  RPU

        7  Cortex-R5 #0 (Halted)

        8  Cortex-R5 #1 (Lock Step Mode)

     9  APU

       10* Cortex-A53 #0 (External debug access is disabled)

       11  Cortex-A53 #1 (No Power)

       12  Cortex-A53 #2 (Running)

       13  Cortex-A53 #3 (Running)

xsdb% ta

  1  PS TAP

     2  PMU

        3  MicroBlaze PMU (Sleeping. No clock)

     4  PL

  5  PSU

     6  RPU

        7  Cortex-R5 #0 (Halted)

        8  Cortex-R5 #1 (Lock Step Mode)

     9  APU

       10* Cortex-A53 #0 (APB AP transaction error, DAP status 0x30000021)

       11  Cortex-A53 #1 (Running)

       12  Cortex-A53 #2 (Running)

       13  Cortex-A53 #3 (No Power)

xsdb%

  • Could you share the exact commit you are on for TF-A

          Merge "docs(changelog): changelog for v2.14 release" into integration · ARM-software/arm-trusted-firmware@1d5aa93

  • Could you share the build configuration for the platform (all build-flags)

           make -j20 RESET_TO_BL31=1 PLAT=zynqmp bl31 IPI_CRC_CHECK=1 DEBUG=1  

           In DEBUG mode ZynqMP will run DDR start address: 0x1000, without debug it will run OCM start address 0x00000000fffea000

  • Would it be possible to share the BL31 logs (so we can see the crash register logs) and Linux logs (just to correlate) as well as bl31.dump/bl31.map file for the failure case (If you can share for the working case as well, that would be great)

             Attached are the working and non-working dumps and logs for NUMA_AWARE_PER_CPU with DDR and OCM. In the working case, LTO is disabled.

             CPU state with debugger with OCM BL31.elf:

             xsdb% ta

  1  PS TAP

     2  PMU

        3  MicroBlaze PMU (Sleeping. No clock)

     4  PL

  5  PSU

     6  RPU

        7  Cortex-R5 #0 (Halted)

        8  Cortex-R5 #1 (Lock Step Mode)

     9  APU

       10* Cortex-A53 #0 (Running)

       11  Cortex-A53 #1 (No Power)

       12  Cortex-A53 #2 (Power On Reset)

       13  Cortex-A53 #3 (Running)

xsdb% Info: Cortex-A53 #2 (target 12) Stopped at 0x0 (Cannot resume. APB AP transaction error, DAP status 0x30000021)

xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power)

xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power)

xsdb% Info: Cortex-A53 #2 (target 12) Stopped at 0x0 (Cannot resume. APB AP transaction error, DAP status 0x30000021)

xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power)

xsdb% Info: Cortex-A53 #0 (target 10) Stopped at 0xfffea194 (Reset Catch)

xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power)

xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power)

xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power)

xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power)

xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power)

xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power)

xsdb% Info: Cortex-A53 #1 (target 11) Running (No Power)

xsdb% ta

  1  PS TAP

     2  PMU

        3  MicroBlaze PMU (Sleeping. No clock)

     4  PL

  5  PSU

     6  RPU

        7  Cortex-R5 #0 (Halted)

        8  Cortex-R5 #1 (Lock Step Mode)

     9  APU

       10* Cortex-A53 #0 (Reset Catch, EL3(S)/A64)

       11  Cortex-A53 #1 (No Power)

       12  Cortex-A53 #2 (Running)

       13  Cortex-A53 #3 (Running)

xsdb% Info: Cortex-A53 #2 (target 12) Running (No Power)

 

Regards,

Rohit

 

 

From: Kummari, Prasad <Prasad.Kummari@amd.com>
Date: Tuesday, 6 January 2026 at 06:34
To: Rohit Mathew <Rohit.Mathew@arm.com>, Sammit Joshi <Sammit.Joshi@arm.com>, scan-admin--- via TF-A <tf-a@lists.trustedfirmware.org>
Cc: Belsare, Akshay <akshay.belsare@amd.com>, Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.com>, Simek, Michal <michal.simek@amd.com>
Subject: RE: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry

[AMD Official Use Only - AMD Internal Distribution Only]

 

Hi Mathew,

Yes, if the CPU_IDLE–related configurations listed below are removed while compiling the kernel image, the hang is no longer observed.

Configs:

CONFIG_CPU_IDLE=y

CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y

CONFIG_CPU_IDLE_GOV_MENU=y

CONFIG_DT_IDLE_STATES=y

Observed Behavior

  • System later triggers an Unhandled Exception in EL3 CPUs enter psci_cpu_suspend(). Debug prints confirm execution reaches the suspend path.
  • The affected CPUs enter Reset Catch at EL3 and hang at the bl31_warm_entrypoint()->el3_entrypoint_common pc.

 

Regards,

Prasad.

 

 

From: Rohit Mathew <Rohit.Mathew@arm.com>
Sent: Tuesday, January 6, 2026 4:37 AM
To: Kummari, Prasad <Prasad.Kummari@amd.com>; Sammit Joshi <Sammit.Joshi@arm.com>; scan-admin--- via TF-A <tf-a@lists.trustedfirmware.org>
Cc: Belsare, Akshay <akshay.belsare@amd.com>; Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.com>; Simek, Michal <michal.simek@amd.com>
Subject: Re: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry

 

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

 

Hi Prasad,

 

 

Thanks for your mail. I’m trying to reproduce this at our end. Could I check if you see the boot progressing to shell without the CPU idle dependencies?

 

Regards,

Rohit

 

From: Kummari, Prasad <Prasad.Kummari@amd.com>
Date: Monday, 5 January 2026 at 18:04
To: Sammit Joshi <Sammit.Joshi@arm.com>, Rohit Mathew <Rohit.Mathew@arm.com>, scan-admin--- via TF-A <tf-a@lists.trustedfirmware.org>
Cc: Belsare, Akshay <akshay.belsare@amd.com>, Bollapalli, Maheedhar Sai <MaheedharSai.Bollapalli@amd.com>, Simek, Michal <michal.simek@amd.com>
Subject: ZynqMP regression with NUMA_AWARE_PER_CPU changes with ENABLED_LTO : Linux runtime hang due to EL3 re-entry

[AMD Official Use Only - AMD Internal Distribution Only]

 

Hello Maintainers,

 

We are observing a reproducible runtime regression on the ZynqMP (Cortex-A53) platform after enabling LTO (ENABLED_LTO=1) and merging the changes from the topic NUMA_AWARE_PER_CPU into our integration branch (https://github.com/ARM-software/arm-trusted-firmware/commit/7303319b3823e9e33748d963e9173f3678aba4da).

 

Summary of the issue

  1. Baseline behavior
    • Platform: ZynqMP (Cortex-A53)
    • Configuration: ENABLED_LTO=1
    • Without NUMA_AWARE_PER_CPU: Linux boots and runs stably
    • After merging NUMA_AWARE_PER_CPU, Linux boots but hangs during runtime
    • During the hang, CPUs are observed to unexpectedly re-enter EL3
    • Reverting the NUMA_AWARE_PER_CPU changes restores stable Linux execution
    • The issue is reproducible only when NUMA_AWARE_PER_CPU is present
    • This clearly identifies NUMA_AWARE_PER_CPU as the regression source
  1. Suspect with LTO
    • With NUMA_AWARE_PER_CPU enabled, LTO breaks the per-CPU base calculation
    • BL31 contains hand-written assembly that relies on linker-script symbols (e.g., per-CPU section boundaries)
    • Under LTO, symbol placement and retention are no longer guaranteed in the same way, leading to incorrect per-CPU base computation
    • This results in corrupted per-CPU data and subsequent erroneous PSCI suspend behavior (EL3 re-entry)
  1. CPU idle dependency
    • The following kernel configuration options are enabled:
    • CONFIG_CPU_IDLE=y
    • CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y
    • CONFIG_CPU_IDLE_GOV_MENU=y
    • CONFIG_DT_IDLE_STATES=y
    • This further suggests the issue is triggered during CPU idle / suspend-resume paths, where correct per-CPU state handling is critical

 

Based on the above:

  • This is specific to NUMA_AWARE_PER_CPU combined with LTO
  • The failure mode points to per-CPU base calculation and PSCI state corruption
  • Reverting NUMA_AWARE_PER_CPU fully restores stability on ZynqMP

We wanted to report this issue upstream and seek guidance on:

  • Whether NUMA_AWARE_PER_CPU is expected to be LTO-safe on platforms relying on linker-defined per-CPU sections
  • Or if additional constraints / fixes are required for platforms like ZynqMP

 

We are happy to provide further logs, configuration details, or help to fixes.

 

Regards,
Prasad Kummari