Hi,
I did the test report SDEI to kernel with fatal severity in APEI / CPER while EL3 received SEA(SCR_EL3.EA = 1). Kernel will panic and print calltrace, but this calltrace was not the position where error occured(another word where throw SEA), instead calltrace in ghes.c.
How can SDEI solution let kernel print calltrace at right position?
For issue analysis, the right position calltrace is very useful. For ACPI firmware-first, we set SCR_EL3.EA = 1, although the solution rethrow EA back to kernel will suffer from some problems, but this solution can let kernel print calltrace at right position.
Best Regards, Ming Huang
On 5/25/21 8:08 PM, James Morse via TF-A wrote:
Hi Pali,
I assume the aborts you are ignoring are precise! SError can be imprecise. I guess these are due to some integration issue with the PCIe root-complex. You shouldn't get aborts outside a RAS error. PCIe has DPC/eDPC and AER for signalling RAS errors - I guess these aren't supported on your platform.
To emulte the synchronous/asynchronous exceptions means emulating what the CPU would do if SCR_EL3.EA weren't set. I'm not sure how familiar you are with the CPU's exception model and all its routing/masking controls. A short tour of what is involved: (the numbered references are for DDI0487G.a, they may have moved in your version)
To emulate an exception you need to copy ESR_EL3, SPSR_EL3 and ELR_EL3 to the ESR, SPSR and ELR of the target EL. You need to calculated new PSTATE and PC for SPSR_EL3 and ELR_EL3. The way the CPU does this is described in the psuedocode in the arm-arm. See AArch64.TakeException. This will involve reading SCTLR of the target EL. For Synchronous-External-Abort, if the FnV bit is clear, you should copy FAR_EL3 to FAR of the target EL too.
There is a GPL implementation of some of these bits in the kernel's KVM code, but that is probably of limited use due to the license.
To determine the target EL you need to examine the routing controls set by the lower ELs. Synchronous Exceptions are the easiest as they can't be masked: The routing rules are describe in D1.12.4 "Routing synchronous External Aborts" is a little terse. It glosses over HCR_EL2.TGE that would also route the exception to EL2.
For a synchronous-external-abort triggered by EL1 where HCR_EL2.TERR is not supported, or clear: If the fault was at stage1, it goes to EL1, if the fault was at stage2, it goes to EL2. There is nothing the ESR to tell you which it was as the CPU indicates this with the target EL, which was overridden by SCR_EL3.EA. (The architecture doesn't expect you to be re-injecting exceptions from EL3). Pragmatically the best option is to allow HCR_EL2.AMO to route synchronous exceptions to EL2 too. (this is a big hint that a hypervisor is managing errors for this exception level). If ELR_EL3.S1PTW is set, this is definitely a stage2 fault, EL1 should never see this bit set. (Synchronous Exceptions can't be routed to a lower EL).
Synchronous errors were the easy one. Asynchronous error routing is described in D1.13.1 "Asynchronous exception routing". You want to take note of HCR_EL2.{AMO, TGE}. But it can also be masked by PSTATE.A of the target EL. (See D1.13.2 "Asynchronous exception masking"): If SPSR_EL3.M is the target EL, SPSR_EL3.A would have masked SError. You cannot emulate the exception if this bit is set. If SPSR_EL3.M is lower than the target EL, then you can emulate the exception to the target EL. If SPSR_EL3.M is higher than the target EL, then SError is effectively masked, and you cannot emulate the exception.
(I don't think HPFAR_EL2 needs to be set for these cases, but I'd need to check)
Finally, you still need something to do if you can't emulate the exception. Updating a system log if you have one and rebooting is the only real option.
Thanks,
James
On 25/05/2021 11:08, Pali Rohár wrote:
Hello!
Platform is not ACPI based. PCIe core in some cases sends External Aborts to kernel which needs to be masked/ignored. I have not found a way how to reconfigure PCIE core to not send these aborts.
In mentioned review is a link to kernel list where was discussion about custom kernel handlers to ignore some of EA. But this approach was rejected with information that TF-A should handle these aborts and ignores those which should not be propagated back to kernel.
If I clear SCR_EL3.EA then aborts (including those which should be ignored) are sent to kernel and kernel makes them fatal. So this is not a solution.
If I do not clear SCR_EL3.EA then in TF-A board/platform code I can implement check for aborts which needs to be ignored. But remaining aborts are not delivered to kernel and TF-A makes them fatal. Which is not correct too.
So, what I need, is to route all External Aborts to TF-A, implement logic which ignores specific PCIE aborts and all remaining aborts needs to be propagated back to kernel like if SCR_EL3.EA is clear.
So it means to implement some logic of abort injection.
On Tuesday 25 May 2021 11:00:09 James Morse wrote:
Hi Guys,
Does this platform need external-aborts to be routed to EL3? If not, you can clear SCR_EL3.EA and be done with it. This allows the EL2 OS/Hypervisor to take control of the routing of these exceptions. (which sounds like what you want)
Otherwise: As Soby describes, the choices are SDEI or emulate the exception according to the arm-arm psuedocode as if EL3 weren't implemented. This is best avoided as its difficult to get right: you have to create a new PSTATE for the target exception level, and read the routing controls to work out which exception level that is.
As Achin says, emulating the exception isn't always possible as Asynchronous exceptions can be masked. The hardware does this automatically when it takes an exception (e.g. irq). (Linux unmask it again once its read the CPU state).
This can leave you holding what may be an imprecise-asynchronous-abort in EL3, unable to emulate the exception or proceed without causing any RAS error to become uncontained. If you can't inject the emulated exception, the error still has to be handled at EL3. If this is an ACPI system you can do a soft restart of the normal-world and present the error via ACPI's BERT (boot error record table) which describes an error that happened in a previous life.
If your platform is ACPI firmware-first, using SDEI will make life easier. You still need to handle the 'SDEI masked' case, but it is a lot less likely to happen. Linux only does this over power-management events that (may) disable the MMU.
(EL2 doesn't have any of these problems as errors are almost always contained by stage2, and it has hardware features for injecting asynchronous exceptions, which cope with the masking and deferring)
Thanks,
James
On 25/05/2021 10:08, Achin Gupta wrote:
Hi,
The last time I checked injecting an SError from a higher to lower EL is a bad idea since the latter could be running with SErrors masked.
EL3 could check this before injecting but then there is no consistent contract with the lower EL about reporting of these errors. SDEI does not suffer from the same problem.
+James who knows more from the OS/Hypervisor perspective.
cheers, Achin
*From:* TF-A tf-a-bounces@lists.trustedfirmware.org on behalf of Soby Mathew via TF-A tf-a@lists.trustedfirmware.org *Sent:* 25 May 2021 09:59 *To:* Pali Rohár pali@kernel.org *Cc:* kabel@kernel.org kabel@kernel.org; tf-a@lists.trustedfirmware.org tf-a@lists.trustedfirmware.org *Subject:* Re: [TF-A] Rethrow SError from EL3 to kernel on arm64 [+tf-a list] Hi Pali, There are 2 philosophies for handing SError in the system, kernel first and firmware first. Assuming you want to stick with firmware first handling (i.e scr_el3.ea is set to 1), then as you mentioned, there are 2 ways to notify the kernel for delegating the error handling: SDEI and SError injection back to kernel. Upstream TF-A only supports SDEI at the moment.
For SError injection back to lower EL, you have to setup the hardware state via software at higher EL in such a way that it appears that the fault was taken to the exception vector at the lower exception level. The pseudocode function AArch64.TakeException() in ARM ARM shows the behavior when the PE takes an exception to an Exception level using AArch64 in Non-debug state. This behaviour has to replicated and it involves the higher EL setting up the PSTATE registers correctly and values in other registers for the lower EL (spsr, elr and fault syndrome registers) and jumping to the right offset point to by the vbar_elx of the lower EL. To the lower EL is appears as a SError has triggered at its exception vector and it can proceed with the fault handling.
Best Regards Soby Mathew
-----Original Message----- From: Pali Rohár pali@kernel.org Sent: Monday, May 24, 2021 6:07 PM To: Soby Mathew Soby.Mathew@arm.com Subject: Rethrow SError from EL3 to kernel on arm64
Hello Soby!
I have found following discussion in Armada 3720 PCIe SError issue: https://review.trustedfirmware.org/c/TF-A/trusted-firmware-
https://review.trustedfirmware.org/c/TF-A/trusted-firmware-
a/+/1541/comment/ca882427_d142bde2/
TF-A on Armada 3720 redirects all SErrors to EL3 and panic in TF-A handler. You wrote in that discussion:
Ideally you need to signal the SError back to kernel from EL3 using SDEI or inject the SError to the lower EL and the kernel can decide to die or not.
And I would like to ask you, could you help me with implementation of this SError rethrow functionality? Because I have absolutely no idea how to do it and catching all SErrors in EL3 is causing issues because some of them can be handled and recovered by kernel.
-- TF-A mailing list TF-A@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-a https://lists.trustedfirmware.org/mailman/listinfo/tf-a
Hello,
On 28/05/2021 13:10, Ming Huang wrote:
I did the test report SDEI to kernel with fatal severity in APEI / CPER while EL3 received SEA(SCR_EL3.EA = 1). Kernel will panic and print calltrace, but this calltrace was not the position where error occured(another word where throw SEA), instead calltrace in ghes.c.
This used to work in the cases where it was possible, but the stack tracing stuff has been changed a little over the recent months.
You didn't mention a kernel version.
How can SDEI solution let kernel print calltrace at right position?
The error was fatal, if the physical address was memory then its just chance that process-A was using it not process-B.
That said, the SDEI entry code in the kernel does try to set the records up to allow the stack tracer to walk onto another stack - but this isn't always possible: The stack tracer needs the frame records to be present and correct, if you took a exception 'x29' needs to be the current frame pointer, but this is only guaranteed at function boundaries. If you took an exception during the functions prologue or epilogue, the values seen by the stack tracer will be inconsistent.
For arm64, linux can't provide a 'reliable' stack trace over an exception, but it does provide a best effort.
For issue analysis, the right position calltrace is very useful. For ACPI firmware-first, we set SCR_EL3.EA = 1, although the solution rethrow EA back to kernel will suffer from some problems, but this solution can let kernel print calltrace at right position.
If you see a complete stacktrace for synchronous external abort delivered directly to the kernel, but not via EL3 and back into SDEI, its likely a problem the kernel has stepping between stacks. SDEI always has to do this, external aborts never do this.
Which kernel version do you see this? (A report to linux-arm-kernel@lists.infradead.org would help too)
Thanks,
James
On 6/1/21 6:57 PM, James Morse wrote:
Hello,
On 28/05/2021 13:10, Ming Huang wrote:
I did the test report SDEI to kernel with fatal severity in APEI / CPER while EL3 received SEA(SCR_EL3.EA = 1). Kernel will panic and print calltrace, but this calltrace was not the position where error occured(another word where throw SEA), instead calltrace in ghes.c.
This used to work in the cases where it was possible, but the stack tracing stuff has been changed a little over the recent months.
You didn't mention a kernel version.
How can SDEI solution let kernel print calltrace at right position?
The error was fatal, if the physical address was memory then its just chance that process-A was using it not process-B.
That said, the SDEI entry code in the kernel does try to set the records up to allow the stack tracer to walk onto another stack - but this isn't always possible: The stack tracer needs the frame records to be present and correct, if you took a exception 'x29' needs to be the current frame pointer, but this is only guaranteed at function boundaries. If you took an exception during the functions prologue or epilogue, the values seen by the stack tracer will be inconsistent.
For arm64, linux can't provide a 'reliable' stack trace over an exception, but it does provide a best effort.
For issue analysis, the right position calltrace is very useful. For ACPI firmware-first, we set SCR_EL3.EA = 1, although the solution rethrow EA back to kernel will suffer from some problems, but this solution can let kernel print calltrace at right position.
If you see a complete stacktrace for synchronous external abort delivered directly to the kernel, but not via EL3 and back into SDEI, its likely a problem the kernel has stepping between stacks. SDEI always has to do this, external aborts never do this.
Which kernel version do you see this? (A report to linux-arm-kernel@lists.infradead.org would help too)
The kernel version: 5.8.0
In my mind, ELR_EL3 and x29 should report to kernel via APEI/CPER for kernel to print calltrace at error position. But there is no suited table in APEI/CPER to report this.
Best Regards, Ming Huang
Thanks,
James
On 02/06/2021 07:11, Ming Huang wrote:> On 6/1/21 6:57 PM, James Morse wrote:>> On 28/05/2021 13:10, Ming Huang wrote>>> I did the test report SDEI to kernel with fatal severity in APEI / CPER while EL3 received>>> SEA(SCR_EL3.EA = 1). Kernel will panic and print calltrace, but this calltrace was not the>>> position where error occured(another word where throw SEA), instead calltrace in ghes.c.>>>> This used to work in the cases where it was possible, but the stack tracing stuff has been>> changed a little over the recent months.>>>> You didn't mention a kernel version.>>>>>>> How can SDEI solution let kernel print calltrace at right position?>>>> The error was fatal, if the physical address was memory then its just chance that>> process-A was using it not process-B.>>>> That said, the SDEI entry code in the kernel does try to set the records up to allow the>> stack tracer to walk onto another stack - but this isn't always possible:>> The stack tracer needs the frame records to be present and correct, if you took a>> exception 'x29' needs to be the current frame pointer, but this is only guaranteed at>> function boundaries. If you took an exception during the functions prologue or epilogue,>> the values seen by the stack tracer will be inconsistent.>>>> For arm64, linux can't provide a 'reliable' stack trace over an exception, but it does>> provide a best effort.>>>>>>> For issue analysis, the right position calltrace is very useful. For ACPI firmware-first,>>> we set SCR_EL3.EA = 1, although the solution rethrow EA back to kernel will suffer from some>>> problems, but this solution can let kernel print calltrace at right position.>>>> If you see a complete stacktrace for synchronous external abort delivered directly to the>> kernel, but not via EL3 and back into SDEI, its likely a problem the kernel has stepping>> between stacks. SDEI always has to do this, external aborts never do this.>>>> Which kernel version do you see this?>> (A report to linux-arm-kernel@lists.infradead.org would help too)> > The kernel version: 5.8.0 is not a kernel.org stable kernel.
Works for me: | Hardware name: Foundation-v8A (DT) | Call trace: | dump_backtrace+0x0/0x1b0 | show_stack+0x18/0x24 | dump_stack+0xc0/0x11c | ghes_in_nmi_queue_one_entry+0x138/0x2f0 | ghes_sdei_normal_callback+0x30/0x6c | sdei_event_handler+0x60/0x1d4 | __sdei_handler+0xc4/0x220 | __sdei_asm_handler+0xbc/0x168 | arch_cpu_idle+0xc/0x20 | cpu_startup_entry+0x24/0x80 | secondary_start_kernel+0x138/0x184 | root@localhost:/sys/kernel/debug# uname -r | 5.8.0-00007-g0db1a507bbda-dirty
In my mind, ELR_EL3 and x29 should report to kernel via APEI/CPER for kernel to print calltrace at error position.
That is not possible.
The kernel can only start a stacktrace from the current location. If your event is synchronous, the kernel should be able to chain one stack onto another, as it does above. If your event is asynchronous, the stack trace is meaningless.
But there is no suited table in APEI/CPER to report this.
See the UEFI spec (where CPER is defined): Table N-26 ARMv8 AArch64 GPRs (Type 4). But, all you can hope to get for populating all this is the kernel print it out. The kernel isn't going to do anything with it.
Thanks,
James
On 6/2/21 10:42 PM, James Morse wrote:
The kernel version: 5.8.0 is not a kernel.org stable kernel.
Works for me: | Hardware name: Foundation-v8A (DT) | Call trace: | dump_backtrace+0x0/0x1b0 | show_stack+0x18/0x24 | dump_stack+0xc0/0x11c | ghes_in_nmi_queue_one_entry+0x138/0x2f0 | ghes_sdei_normal_callback+0x30/0x6c | sdei_event_handler+0x60/0x1d4 | __sdei_handler+0xc4/0x220 | __sdei_asm_handler+0xbc/0x168 | arch_cpu_idle+0xc/0x20 | cpu_startup_entry+0x24/0x80 | secondary_start_kernel+0x138/0x184 | root@localhost:/sys/kernel/debug# uname -r | 5.8.0-00007-g0db1a507bbda-dirty
How can you print the call trace rightly? In my test, I can't get call trace rightly with reporting CPER + SDEI.
Linux version 5.10.0 [root@m1rootfs /]$devmem 0 ERROR: Excepton received on 0x81000000, spsr_el3:60001000,reason:0 esr_el3:0xbe000011 [ 129.772039] sdei: sdei_event_handler this handler event: 806 [ 129.772519] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3 [ 129.772768] {2}[Hardware Error]: event severity: fatal [ 129.772962] {2}[Hardware Error]: Error 0, type: fatal [ 129.773154] {2}[Hardware Error]: section_type: general processor error [ 129.773395] {2}[Hardware Error]: processor_type: 2, ARM [ 129.773503] {2}[Hardware Error]: processor_isa: 4, ARM A64 [ 129.773605] {2}[Hardware Error]: error_type: 0x00 [ 129.773725] {2}[Hardware Error]: operation: 0, unknown or generic [ 129.773833] {2}[Hardware Error]: processor_id: 0x0000000000000000 [ 129.773943] Kernel panic - not syncing: Fatal hardware error! [ 129.774060] CPU: 0 PID: 201 Comm: devmem Not tainted 5.10.0+ #25 [ 129.774195] Hardware name: Default Default/Default, BIOS 1.2.M1.AL.E.050.00 06/02/2021 [ 129.774273] Call trace: [ 129.774378] dump_backtrace+0x0/0x1f0 [ 129.774459] show_stack+0x24/0x70 [ 129.774543] dump_stack+0xbc/0x114 [ 129.774621] panic+0x158/0x364 [ 129.774723] __raw_spin_lock_irqsave.constprop.0+0x0/0xa0 [ 129.774820] ghes_in_nmi_queue_one_entry+0x204/0x2fc [ 129.774917] ghes_sdei_normal_callback+0x58/0xc0 [ 129.775005] sdei_event_handler+0x50/0xe8 [ 129.775090] _sdei_handler+0x8c/0x160 [ 129.775167] __sdei_handler+0x28/0x50 [ 129.775265] __sdei_asm_handler+0xbc/0x174 [ 129.779672] SMP: stopping secondary CPUs [ 129.779766] Kernel Offset: disabled
Best Regards, Ming Huang
In my mind, ELR_EL3 and x29 should report to kernel via APEI/CPER for kernel to print calltrace at error position.
That is not possible.
The kernel can only start a stacktrace from the current location. If your event is synchronous, the kernel should be able to chain one stack onto another, as it does above. If your event is asynchronous, the stack trace is meaningless.
But there is no suited table in APEI/CPER to report this.
See the UEFI spec (where CPER is defined): Table N-26 ARMv8 AArch64 GPRs (Type 4). But, all you can hope to get for populating all this is the kernel print it out. The kernel isn't going to do anything with it.
Thanks,
James
Hello,
On 23/06/2021 02:21, Ming Huang wrote:
On 6/2/21 10:42 PM, James Morse wrote:
The kernel version: 5.8.0 is not a kernel.org stable kernel.
Works for me: | Hardware name: Foundation-v8A (DT) | Call trace: | dump_backtrace+0x0/0x1b0 | show_stack+0x18/0x24 | dump_stack+0xc0/0x11c | ghes_in_nmi_queue_one_entry+0x138/0x2f0 | ghes_sdei_normal_callback+0x30/0x6c | sdei_event_handler+0x60/0x1d4 | __sdei_handler+0xc4/0x220 | __sdei_asm_handler+0xbc/0x168 | arch_cpu_idle+0xc/0x20 | cpu_startup_entry+0x24/0x80 | secondary_start_kernel+0x138/0x184 | root@localhost:/sys/kernel/debug# uname -r | 5.8.0-00007-g0db1a507bbda-dirty
How can you print the call trace rightly? In my test, I can't get call trace rightly with reporting CPER + SDEI.
Linux version 5.10.0 [root@m1rootfs /]$devmem 0 ERROR: Excepton received on 0x81000000, spsr_el3:60001000,reason:0 esr_el3:0xbe000011 [ 129.772039] sdei: sdei_event_handler this handler event: 806 [ 129.772519] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3 [ 129.772768] {2}[Hardware Error]: event severity: fatal [ 129.772962] {2}[Hardware Error]: Error 0, type: fatal [ 129.773154] {2}[Hardware Error]: section_type: general processor error [ 129.773395] {2}[Hardware Error]: processor_type: 2, ARM [ 129.773503] {2}[Hardware Error]: processor_isa: 4, ARM A64 [ 129.773605] {2}[Hardware Error]: error_type: 0x00 [ 129.773725] {2}[Hardware Error]: operation: 0, unknown or generic [ 129.773833] {2}[Hardware Error]: processor_id: 0x0000000000000000 [ 129.773943] Kernel panic - not syncing: Fatal hardware error! [ 129.774060] CPU: 0 PID: 201 Comm: devmem Not tainted 5.10.0+ #25 [ 129.774195] Hardware name: Default Default/Default, BIOS 1.2.M1.AL.E.050.00 06/02/2021 [ 129.774273] Call trace: [ 129.774378] dump_backtrace+0x0/0x1f0 [ 129.774459] show_stack+0x24/0x70 [ 129.774543] dump_stack+0xbc/0x114 [ 129.774621] panic+0x158/0x364 [ 129.774723] __raw_spin_lock_irqsave.constprop.0+0x0/0xa0 [ 129.774820] ghes_in_nmi_queue_one_entry+0x204/0x2fc [ 129.774917] ghes_sdei_normal_callback+0x58/0xc0 [ 129.775005] sdei_event_handler+0x50/0xe8 [ 129.775090] _sdei_handler+0x8c/0x160 [ 129.775167] __sdei_handler+0x28/0x50 [ 129.775265] __sdei_asm_handler+0xbc/0x174
This _is_ the call stack. What else were you expecting to see?
Your SPSR_EL3 debug message shows the exception came from EL0. The kernel has not found any kernel stack frames below __sdei_asm_handler, because there aren't any! The kernel does not produce stack traces for user-space.
[ 129.779672] SMP: stopping secondary CPUs [ 129.779766] Kernel Offset: disabled
Thanks,
James
Hi James,
Appologize for reply so late. I miss out your email.
As your hint, I make a einj ACPI table with a bug for tiggering EA in kernel and then get the call trace that we expect. __einj_error_trigger below __sdei_asm_handler, indicate where error occured.
Thank you very much!
My test log: ---------------------------------------------- [root@rootfs einj]$echo 0x8A2000000 > param1 [root@rootfs einj]$echo 0xfffffffffffff000 > param2 [root@rootfs einj]$echo 0x1 > flags [root@rootfs einj]$echo 0x08 > error_type [root@rootfs einj]$echo 1 > error_inject INFO: Core[ER1R](0OxR810:100 00) re ceiEed ras intr=115, cnt=1. on 0x181000000, spsr_el3:62400009,reason:1 esr_el3:0Einj ErrType:8 Flags:1, ApicId:0 DDDR Einj Addr:8A2000000 Range:FFFFFFFFFFFFF000 nknInvalid & clear cache(8A2000000) [64](0x181000000) received ras intr=0, cnt=0. CPU RAS mm handler: EventId=C4000048
[ 199.563096] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3 [ 199.563228] {2}[Hardware Error]: event severity: fatal [ 199.563317] {2}[Hardware Error]: Error 0, type: fatal [ 199.563379] {2}[Hardware Error]: section_type: general processor error [ 199.563439] {2}[Hardware Error]: processor_type: 2, ARM [ 199.563502] {2}[Hardware Error]: processor_isa: 4, ARM A64 [ 199.563556] {2}[Hardware Error]: error_type: 0x00 [ 199.563618] {2}[Hardware Error]: operation: 0, unknown or generic [ 199.563679] {2}[Hardware Error]: processor_id: 0x0000000000000040 [ 199.563735] Kernel panic - not syncing: Fatal hardware error! [ 199.563890] CPU: 2 PID: 236 Comm: bash Tainted: G E 5.10.23 #1 [ 199.563958] Hardware name: Default Default/Default, BIOS 1.2.AL.E.105.00 08/13/2021 [ 199.564000] Call trace: [ 199.564047] dump_backtrace+0x0/0x1e0 [ 199.564091] show_stack+0x1c/0x24 [ 199.564138] dump_stack+0xcc/0x120 [ 199.564178] panic+0x154/0x360 [ 199.564226] __ghes_panic+0x7c/0x80 [ 199.564283] ghes_in_nmi_queue_one_entry+0x1fc/0x2f4 [ 199.564335] ghes_sdei_critical_callback+0x50/0xb0 [ 199.564385] sdei_event_handler+0x3c/0xb0 [ 199.564429] _sdei_handler+0x88/0x120 [ 199.564471] __sdei_handler+0x24/0x50 [ 199.564519] __sdei_asm_handler+0xbc/0x15c [ 199.564570] __einj_error_trigger+0x7c/0x3d0 [einj] [ 199.564621] __einj_error_inject+0x24c/0x2ac [einj] [ 199.564668] error_inject_set+0xcc/0x134 [einj] [ 199.564715] simple_attr_write+0xfc/0x130 [ 199.564759] debugfs_attr_write+0x50/0x90 [ 199.564802] vfs_write+0xd0/0x290 [ 199.564846] ksys_write+0x70/0x100 [ 199.564891] __arm64_sys_write+0x20/0x30 [ 199.564939] el0_svc_common.constprop.0+0x70/0x170 [ 199.564986] do_el0_svc+0x74/0x90 [ 199.565031] el0_svc+0x1c/0x30 [ 199.565079] el0_sync_handler+0xa8/0xac [ 199.565121] el0_sync+0x140/0x180 [ 199.565172] SMP: stopping secondary CPUs [ 199.565224] Kernel Offset: disabled [ 199.565278] CPU features: 0x9850817,7a60aa38 [ 199.565356] Memory Limit: none INFO: PSCI Power Domain Map: INFO: Domain Node : Level 1, parent_node -1, State OFF (0x2) INFO: Domain Node : Level 1, parent_node -1, State OFF (0x2) ----------------------------------------------
On 6/23/21 6:06 PM, James Morse wrote:
Hello,
On 23/06/2021 02:21, Ming Huang wrote:
On 6/2/21 10:42 PM, James Morse wrote:
The kernel version: 5.8.0 is not a kernel.org stable kernel.
Works for me: | Hardware name: Foundation-v8A (DT) | Call trace: | dump_backtrace+0x0/0x1b0 | show_stack+0x18/0x24 | dump_stack+0xc0/0x11c | ghes_in_nmi_queue_one_entry+0x138/0x2f0 | ghes_sdei_normal_callback+0x30/0x6c | sdei_event_handler+0x60/0x1d4 | __sdei_handler+0xc4/0x220 | __sdei_asm_handler+0xbc/0x168 | arch_cpu_idle+0xc/0x20 | cpu_startup_entry+0x24/0x80 | secondary_start_kernel+0x138/0x184 | root@localhost:/sys/kernel/debug# uname -r | 5.8.0-00007-g0db1a507bbda-dirty
How can you print the call trace rightly? In my test, I can't get call trace rightly with reporting CPER + SDEI.
Linux version 5.10.0 [root@m1rootfs /]$devmem 0 ERROR: Excepton received on 0x81000000, spsr_el3:60001000,reason:0 esr_el3:0xbe000011 [ 129.772039] sdei: sdei_event_handler this handler event: 806 [ 129.772519] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3 [ 129.772768] {2}[Hardware Error]: event severity: fatal [ 129.772962] {2}[Hardware Error]: Error 0, type: fatal [ 129.773154] {2}[Hardware Error]: section_type: general processor error [ 129.773395] {2}[Hardware Error]: processor_type: 2, ARM [ 129.773503] {2}[Hardware Error]: processor_isa: 4, ARM A64 [ 129.773605] {2}[Hardware Error]: error_type: 0x00 [ 129.773725] {2}[Hardware Error]: operation: 0, unknown or generic [ 129.773833] {2}[Hardware Error]: processor_id: 0x0000000000000000 [ 129.773943] Kernel panic - not syncing: Fatal hardware error! [ 129.774060] CPU: 0 PID: 201 Comm: devmem Not tainted 5.10.0+ #25 [ 129.774195] Hardware name: Default Default/Default, BIOS 1.2.M1.AL.E.050.00 06/02/2021 [ 129.774273] Call trace: [ 129.774378] dump_backtrace+0x0/0x1f0 [ 129.774459] show_stack+0x24/0x70 [ 129.774543] dump_stack+0xbc/0x114 [ 129.774621] panic+0x158/0x364 [ 129.774723] __raw_spin_lock_irqsave.constprop.0+0x0/0xa0 [ 129.774820] ghes_in_nmi_queue_one_entry+0x204/0x2fc [ 129.774917] ghes_sdei_normal_callback+0x58/0xc0 [ 129.775005] sdei_event_handler+0x50/0xe8 [ 129.775090] _sdei_handler+0x8c/0x160 [ 129.775167] __sdei_handler+0x28/0x50 [ 129.775265] __sdei_asm_handler+0xbc/0x174
This _is_ the call stack. What else were you expecting to see?
Your SPSR_EL3 debug message shows the exception came from EL0. The kernel has not found any kernel stack frames below __sdei_asm_handler, because there aren't any! The kernel does not produce stack traces for user-space.
[ 129.779672] SMP: stopping secondary CPUs [ 129.779766] Kernel Offset: disabled
Thanks,
James
tf-a@lists.trustedfirmware.org
-
James Morse -
Ming Huang