Hi, Govindrj R,

Thanks for providing the points where I can dig into more. Yes, I took a look at the code related to PSCI and I realized that I may need to add similar operation like psci_cpu_on_start. (we may want opteed_cpu_on_finish_handler() to be finally called on TF-A?) But I have no confidence and I am asking about what I really do. Depends on the CPU enable method, SMP boot flow seems to differ. As for spintable method, I guess that I need to create new interface which triggers optee context initialization on secondary cores.

Based on this, what I'm curious is if I need to add something to linux kernel or only need to implement something in TF-A or donot need to add anything... (and where I need to implement it if I need to create something) If there is a reference code, it will be the best :)

If I understand wrongly or miss somthing else, please let me know.

Thanks, --Bin

2023년 8월 17일 (목) 오전 7:23, Govindraj Raja Govindraj.Raja@arm.com님이 작성:

Hi Bin,

Did you check what psci_cpu_on_start​ is doing I would expect you have similar operations when you want to bypass the psci cpu enable method.

Few things I can notice from code which you may have to explore:

• Perform cache maintanence ahead of reading the target CPU state to

ensure that the data is not stale.

• Platform specific power on.
• Store the re-entry information for the non-secure world - context

management for CPU.

--

Thanks,

Govindraj R

*From:* 빈성욱 via TF-A tf-a@lists.trustedfirmware.org *Sent:* Wednesday, August 16, 2023 01:12 *To:* tf-a@lists.trustedfirmware.org tf-a@lists.trustedfirmware.org *Subject:* [TF-A] Question about Secondary CPU boot with spintable

Hi

I ported TF-A and OPTEE on multi-core SoC, and I checked that xtest passed on CPU0 without failure. However, I'm failing in secondary CPU boot, and I'd like to get some help.

I'm using ARMv8, and I'm using spintable as CPU enable method (not using PSCI FW). TF-A is in EL3.

What I did ...

1. I released secondary CPU reset from bl31_early_platform_setup2() during

primary CPU cold booting - set secondary CPU's RVBAR to bl31_entrypoint - release secondary CPU reset 2. I implemeneted plat_secondary_cold_boot_setup(), which is spining until linux kernel updates cpu-release-addr. Once linux kernel updates cpu-release-addr, then it changes EL to EL1 and jumps to secondary_startup() in linux 3. But secondary core goes to panic. It seems to happen when linux invokes optee_probe()

Should I initialize some contexts (e.g. optee ..)? If so, where should I implement the corresponding codes? I'm wondering if I'm missing something, and I'd like to get some advice

Thanks, --Bin

Hi Bin

Like Jeremy says, I would strongly recommend using PSCI, even if this is backed by a spintable in your platform implementation. This is what some upstream platforms do (e.g. RPI3).

If your platform does not automatically release secondary CPUs at the same time as releasing the primary and you've programmed the secondary CPU's RVBAR, then there's no need to implement plat_secondary_cold_boot_setup() (or you could just provide a dummy implementation). Your implementation of pwr_domain_on could release the secondary CPUs from reset the first time it is called. On subsequent calls, you could release the CPU from the spintable. Your implementation of pwr_domain_pwr_down_wfi could put the CPUs into the spintable.

The reset design docs here might help: https://trustedfirmware-a.readthedocs.io/en/latest/design/reset-design.htm

Regards

Dan.

-----Original Message----- From: Jeremy Linton via TF-A tf-a@lists.trustedfirmware.org Sent: Friday, August 18, 2023 1:00 AM To: 빈성욱 sunguk.bin@gmail.com; Govindraj Raja Govindraj.Raja@arm.com Cc: tf-a@lists.trustedfirmware.org Subject: [TF-A] Re: Question about Secondary CPU boot with spintable

Hi,

On 8/16/23 22:18, 빈성욱 via TF-A wrote:

...

Hi, Govindrj R,

Thanks for providing the points where I can dig into more. Yes, I took a look at the code related to PSCI and I realized that I may need to add similar operation like psci_cpu_on_start. (we may want opteed_cpu_on_finish_handler() to be finally called on TF-A?) But I have no confidence and I am asking about what I really do. Depends on the CPU enable method, SMP boot flow seems to differ. As for spintable method, I guess that I need to create new interface which triggers optee context initialization on secondary cores.

Based on this, what I'm curious is if I need to add something to linux kernel or only need to implement something in TF-A or donot need to add anything... (and where I need to implement it if I need to create something) If there is a reference code, it will be the best :)

I noticed this in passing, and its not entirely apparent to me why you selected a spintable method to release your secondary cores . The arm64, acpi_parking_protocol_ops and smp_spin_table_ops exist as fallbacks for hardware that doesn't implement EL3. AFAIK, Its _strongly_ encouraged that if the machine can support PSCI it should be used, even if the PSCI calls end up forwarding the requests somewhere else.

...

If I understand wrongly or miss somthing else, please let me know.

Thanks, --Bin

2023년 8월 17일 (목) 오전 7:23, Govindraj Raja

Govindraj.Raja@arm.com님이 작성:

...

...

Hi Bin,

Did you check what psci_cpu_on_start​ is doing I would expect you have similar operations when you want to bypass the psci cpu enable method.

Few things I can notice from code which you may have to explore:

- Perform cache maintanence ahead of reading the target CPU state to
ensure that the data is not stale.
- Platform specific power on.
- Store the re-entry information for the non-secure world - context
management for CPU.

--

Thanks,

Govindraj R

*From:* 빈성욱 via TF-A tf-a@lists.trustedfirmware.org *Sent:* Wednesday, August 16, 2023 01:12 *To:* tf-a@lists.trustedfirmware.org tf-a@lists.trustedfirmware.org *Subject:* [TF-A] Question about Secondary CPU boot with spintable

Hi

I ported TF-A and OPTEE on multi-core SoC, and I checked that xtest passed on CPU0 without failure. However, I'm failing in secondary CPU boot, and I'd like to get some help.

I'm using ARMv8, and I'm using spintable as CPU enable method (not using PSCI FW). TF-A is in EL3.

What I did ...

1. I released secondary CPU reset from bl31_early_platform_setup2()

during primary CPU cold booting - set secondary CPU's RVBAR to bl31_entrypoint - release secondary CPU reset 2. I implemeneted plat_secondary_cold_boot_setup(), which is spining until linux kernel updates cpu-release-addr. Once linux kernel updates cpu-release-addr, then it changes EL to EL1 and jumps to secondary_startup() in linux 3. But secondary core goes to panic. It seems to happen when linux invokes optee_probe()

Should I initialize some contexts (e.g. optee ..)? If so, where should I implement the corresponding codes? I'm wondering if I'm missing something, and I'd like to get some advice

Thanks, --Bin

-- TF-A mailing list -- tf-a@lists.trustedfirmware.org To unsubscribe send an email to tf-a-leave@lists.trustedfirmware.org

Thanks for all answers. I'll try PSCI, and I'd like to check something more. 1. Should I set COLD_BOOT_SINGLE_CPU to 1? Is this mandatory for PSCI use? 2. I understand that pwr_domain_on() releases its own CPU reset only (e.g. if cpu2, it release cpu2 reset only) Is it correct? 3. Should I set RVBAR to bl31_warm_entrypoint?

Regards, --Bin