Hi Andre,
I'm trying to get CPU offlining work for Linux on the RPi4. In fact, this is working already with current ATF master and the Raspberry kernel 4.19.85. Not it would be "nice" if onlining worked as well:
[ 94.959294] CPU1: shutdown [ 94.959311] psci: CPU1 killed. [ 106.750960] CPU1: failed to come online [ 106.755425] CPU1: failed in unknown state : 0x0
Is this a known limitation? Or am I using the wrong kernel (obviously, it's a downstream one ;) )?
Thanks, Jan
PS: Current RPi firmware requires an explicit "armstub=armstub8.bin" in config.txt, no automatic pickup.
On 24/11/2019 13:21, Jan Kiszka wrote:
Hi,
I'm trying to get CPU offlining work for Linux on the RPi4. In fact, this is working already with current ATF master and the Raspberry kernel 4.19.85. Not it would be "nice" if onlining worked as well:
Indeed.
[ 94.959294] CPU1: shutdown [ 94.959311] psci: CPU1 killed. [ 106.750960] CPU1: failed to come online [ 106.755425] CPU1: failed in unknown state : 0x0
Yeah, I remember there was an issue with that, but I didn't come around to fix that yet. I would have put this as a limitation in the cover letter, but: gerrit ...
Is this a known limitation? Or am I using the wrong kernel (obviously, it's a downstream one ;) )?
I obviously cannot endorse the usage of downstream kernels ;-), but I think I saw this with mainline as well. Do you get any more messages on the console? I think RPI3_RUNTIME_UART defaults to 0 for the RPI4 (more by accident), which should give you TF-A debug messages on PSCI calls. Maybe you can quickly add some, to see where it breaks? I might dig out my RPI4, but only later next week, so you might want to beat me on it, if you have your rig set up already.
Thanks, Jan
PS: Current RPi firmware requires an explicit "armstub=armstub8.bin" in config.txt, no automatic pickup.
Ah, thanks for the heads up. Feel free to send a documentation patch - if you are comfortable with gerrit, that is. Otherwise I would do it later on.
Cheers, Andre.
On 24.11.19 15:19, André Przywara wrote:
On 24/11/2019 13:21, Jan Kiszka wrote:
Hi,
I'm trying to get CPU offlining work for Linux on the RPi4. In fact, this is working already with current ATF master and the Raspberry kernel 4.19.85. Not it would be "nice" if onlining worked as well:
Indeed.
[ 94.959294] CPU1: shutdown [ 94.959311] psci: CPU1 killed. [ 106.750960] CPU1: failed to come online [ 106.755425] CPU1: failed in unknown state : 0x0
Yeah, I remember there was an issue with that, but I didn't come around to fix that yet. I would have put this as a limitation in the cover letter, but: gerrit ...
Is this a known limitation? Or am I using the wrong kernel (obviously, it's a downstream one ;) )?
I obviously cannot endorse the usage of downstream kernels ;-), but I think I saw this with mainline as well.
Is mainline working on the RPi4 already? Well, another topic...
Do you get any more messages on the console? I think RPI3_RUNTIME_UART defaults to 0 for the RPI4 (more by accident), which should give you TF-A debug messages on PSCI calls. Maybe you can quickly add some, to see where it breaks?
I've instrumented the psci cpu off/on paths, and the enabling path after disabling never gets us into psci_warmboot_entrypoint() at least.
I might dig out my RPI4, but only later next week, so you might want to beat me on it, if you have your rig set up already.
I this working on the RPi3? I don't see much deviations between both in this regard.
Thanks, Jan
PS: Current RPi firmware requires an explicit "armstub=armstub8.bin" in config.txt, no automatic pickup.
Ah, thanks for the heads up. Feel free to send a documentation patch - if you are comfortable with gerrit, that is. Otherwise I would do it later on.
Well, gerrit... Let's see whether I'll be able to produce enough changes to be worth that trouble.
Thanks, Jan
On 24.11.19 15:49, Jan Kiszka wrote:
On 24.11.19 15:19, André Przywara wrote:
On 24/11/2019 13:21, Jan Kiszka wrote:
Hi,
I'm trying to get CPU offlining work for Linux on the RPi4. In fact, this is working already with current ATF master and the Raspberry kernel 4.19.85. Not it would be "nice" if onlining worked as well:
Indeed.
[ 94.959294] CPU1: shutdown [ 94.959311] psci: CPU1 killed. [ 106.750960] CPU1: failed to come online [ 106.755425] CPU1: failed in unknown state : 0x0
Yeah, I remember there was an issue with that, but I didn't come around to fix that yet. I would have put this as a limitation in the cover letter, but: gerrit ...
Is this a known limitation? Or am I using the wrong kernel (obviously, it's a downstream one ;) )?
I obviously cannot endorse the usage of downstream kernels ;-), but I think I saw this with mainline as well.
Is mainline working on the RPi4 already? Well, another topic...
Do you get any more messages on the console? I think RPI3_RUNTIME_UART defaults to 0 for the RPI4 (more by accident), which should give you TF-A debug messages on PSCI calls. Maybe you can quickly add some, to see where it breaks?
I've instrumented the psci cpu off/on paths, and the enabling path after disabling never gets us into psci_warmboot_entrypoint() at least.
I might dig out my RPI4, but only later next week, so you might want to beat me on it, if you have your rig set up already.
I this working on the RPi3? I don't see much deviations between both in this regard.
Likely not: rpi3_pwr_domain_off doesn't look to me like it is seriously pulling any plug. And rpi3_pwr_domain_on expect the target to already spin in plat_secondary_cold_boot_setup, in reset state.
Here is a hack to make onlining - apparently - work:
diff --git a/plat/rpi/common/rpi3_pm.c b/plat/rpi/common/rpi3_pm.c index 8c2d070c4..2a6bf076b 100644 --- a/plat/rpi/common/rpi3_pm.c +++ b/plat/rpi/common/rpi3_pm.c @@ -123,6 +123,15 @@ static void rpi3_pwr_domain_off(const psci_power_state_t *target_state) #endif }
+void __dead2 plat_secondary_cold_boot_setup(void); + +static void __dead2 +rpi3_pwr_domain_pwr_down_wfi(const psci_power_state_t *target_state) +{ + disable_mmu_el3(); + plat_secondary_cold_boot_setup(); +} + /******************************************************************************* * Platform handler called when a power domain is about to be turned on. The * mpidr determines the CPU to be turned on. @@ -224,6 +233,7 @@ static void __dead2 rpi3_system_off(void) static const plat_psci_ops_t plat_rpi3_psci_pm_ops = { .cpu_standby = rpi3_cpu_standby, .pwr_domain_off = rpi3_pwr_domain_off, + .pwr_domain_pwr_down_wfi = rpi3_pwr_domain_pwr_down_wfi, .pwr_domain_on = rpi3_pwr_domain_on, .pwr_domain_on_finish = rpi3_pwr_domain_on_finish, .system_off = rpi3_system_off,
Jan
On 24.11.19 16:08, Jan Kiszka wrote:
On 24.11.19 15:49, Jan Kiszka wrote:
On 24.11.19 15:19, André Przywara wrote:
On 24/11/2019 13:21, Jan Kiszka wrote:
Hi,
I'm trying to get CPU offlining work for Linux on the RPi4. In fact, this is working already with current ATF master and the Raspberry kernel 4.19.85. Not it would be "nice" if onlining worked as well:
Indeed.
[ 94.959294] CPU1: shutdown [ 94.959311] psci: CPU1 killed. [ 106.750960] CPU1: failed to come online [ 106.755425] CPU1: failed in unknown state : 0x0
Yeah, I remember there was an issue with that, but I didn't come around to fix that yet. I would have put this as a limitation in the cover letter, but: gerrit ...
Is this a known limitation? Or am I using the wrong kernel (obviously, it's a downstream one ;) )?
I obviously cannot endorse the usage of downstream kernels ;-), but I think I saw this with mainline as well.
Is mainline working on the RPi4 already? Well, another topic...
Do you get any more messages on the console? I think RPI3_RUNTIME_UART defaults to 0 for the RPI4 (more by accident), which should give you TF-A debug messages on PSCI calls. Maybe you can quickly add some, to see where it breaks?
I've instrumented the psci cpu off/on paths, and the enabling path after disabling never gets us into psci_warmboot_entrypoint() at least.
I might dig out my RPI4, but only later next week, so you might want to beat me on it, if you have your rig set up already.
I this working on the RPi3? I don't see much deviations between both in this regard.
Likely not: rpi3_pwr_domain_off doesn't look to me like it is seriously pulling any plug. And rpi3_pwr_domain_on expect the target to already spin in plat_secondary_cold_boot_setup, in reset state.
Here is a hack to make onlining - apparently - work:
diff --git a/plat/rpi/common/rpi3_pm.c b/plat/rpi/common/rpi3_pm.c index 8c2d070c4..2a6bf076b 100644 --- a/plat/rpi/common/rpi3_pm.c +++ b/plat/rpi/common/rpi3_pm.c @@ -123,6 +123,15 @@ static void rpi3_pwr_domain_off(const psci_power_state_t *target_state) #endif }
+void __dead2 plat_secondary_cold_boot_setup(void);
+static void __dead2 +rpi3_pwr_domain_pwr_down_wfi(const psci_power_state_t *target_state) +{
- disable_mmu_el3();
- plat_secondary_cold_boot_setup();
+}
/*******************************************************************************
- Platform handler called when a power domain is about to be turned on. The
- mpidr determines the CPU to be turned on.
@@ -224,6 +233,7 @@ static void __dead2 rpi3_system_off(void) static const plat_psci_ops_t plat_rpi3_psci_pm_ops = { .cpu_standby = rpi3_cpu_standby, .pwr_domain_off = rpi3_pwr_domain_off,
- .pwr_domain_pwr_down_wfi = rpi3_pwr_domain_pwr_down_wfi, .pwr_domain_on = rpi3_pwr_domain_on, .pwr_domain_on_finish = rpi3_pwr_domain_on_finish, .system_off = rpi3_system_off,
Jan
Works well, Jailhouse is happy on the RPi4 about the newly gained PSCI support.
How should we continue with the patch below?
Jan
---
Subject: [PATCH] rpi3/4: Add support for offlining CPUs From: Jan Kiszka jan.kiszka@siemens.com
The hooks were populated but the power down left the CPU in limbo-land. What we need to do - until there is a way to actually power off - is to turn off the MMU and enter the spinning loop as if we were cold-booted. This allows the on-call to pick up the CPU again.
Signed-off-by: Jan Kiszka jan.kiszka@siemens.com --- plat/rpi/common/rpi3_pm.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/plat/rpi/common/rpi3_pm.c b/plat/rpi/common/rpi3_pm.c index 8c2d070c4..2a6bf076b 100644 --- a/plat/rpi/common/rpi3_pm.c +++ b/plat/rpi/common/rpi3_pm.c @@ -123,6 +123,15 @@ static void rpi3_pwr_domain_off(const psci_power_state_t *target_state) #endif }
+void __dead2 plat_secondary_cold_boot_setup(void); + +static void __dead2 +rpi3_pwr_domain_pwr_down_wfi(const psci_power_state_t *target_state) +{ + disable_mmu_el3(); + plat_secondary_cold_boot_setup(); +} + /******************************************************************************* * Platform handler called when a power domain is about to be turned on. The * mpidr determines the CPU to be turned on. @@ -224,6 +233,7 @@ static void __dead2 rpi3_system_off(void) static const plat_psci_ops_t plat_rpi3_psci_pm_ops = { .cpu_standby = rpi3_cpu_standby, .pwr_domain_off = rpi3_pwr_domain_off, + .pwr_domain_pwr_down_wfi = rpi3_pwr_domain_pwr_down_wfi, .pwr_domain_on = rpi3_pwr_domain_on, .pwr_domain_on_finish = rpi3_pwr_domain_on_finish, .system_off = rpi3_system_off, -- 2.16.4
On Mon, 9 Dec 2019 08:41:47 +0100 Jan Kiszka jan.kiszka@web.de wrote:
Hi,
On 24.11.19 16:08, Jan Kiszka wrote:
On 24.11.19 15:49, Jan Kiszka wrote:
On 24.11.19 15:19, André Przywara wrote:
On 24/11/2019 13:21, Jan Kiszka wrote:
Hi,
I'm trying to get CPU offlining work for Linux on the RPi4. In fact, this is working already with current ATF master and the Raspberry kernel 4.19.85. Not it would be "nice" if onlining worked as well:
Indeed.
[ 94.959294] CPU1: shutdown [ 94.959311] psci: CPU1 killed. [ 106.750960] CPU1: failed to come online [ 106.755425] CPU1: failed in unknown state : 0x0
Yeah, I remember there was an issue with that, but I didn't come around to fix that yet. I would have put this as a limitation in the cover letter, but: gerrit ...
Is this a known limitation? Or am I using the wrong kernel (obviously, it's a downstream one ;) )?
I obviously cannot endorse the usage of downstream kernels ;-), but I think I saw this with mainline as well.
Is mainline working on the RPi4 already? Well, another topic...
Do you get any more messages on the console? I think RPI3_RUNTIME_UART defaults to 0 for the RPI4 (more by accident), which should give you TF-A debug messages on PSCI calls. Maybe you can quickly add some, to see where it breaks?
I've instrumented the psci cpu off/on paths, and the enabling path after disabling never gets us into psci_warmboot_entrypoint() at least.
I might dig out my RPI4, but only later next week, so you might want to beat me on it, if you have your rig set up already.
I this working on the RPi3? I don't see much deviations between both in this regard.
Likely not: rpi3_pwr_domain_off doesn't look to me like it is seriously pulling any plug. And rpi3_pwr_domain_on expect the target to already spin in plat_secondary_cold_boot_setup, in reset state.
Here is a hack to make onlining - apparently - work:
diff --git a/plat/rpi/common/rpi3_pm.c b/plat/rpi/common/rpi3_pm.c index 8c2d070c4..2a6bf076b 100644 --- a/plat/rpi/common/rpi3_pm.c +++ b/plat/rpi/common/rpi3_pm.c @@ -123,6 +123,15 @@ static void rpi3_pwr_domain_off(const psci_power_state_t *target_state) #endif }
+void __dead2 plat_secondary_cold_boot_setup(void);
+static void __dead2 +rpi3_pwr_domain_pwr_down_wfi(const psci_power_state_t *target_state) +{
- disable_mmu_el3();
- plat_secondary_cold_boot_setup();
+}
/*******************************************************************************
- Platform handler called when a power domain is about to be turned on. The
- mpidr determines the CPU to be turned on.
@@ -224,6 +233,7 @@ static void __dead2 rpi3_system_off(void) static const plat_psci_ops_t plat_rpi3_psci_pm_ops = { .cpu_standby = rpi3_cpu_standby, .pwr_domain_off = rpi3_pwr_domain_off,
- .pwr_domain_pwr_down_wfi = rpi3_pwr_domain_pwr_down_wfi, .pwr_domain_on = rpi3_pwr_domain_on, .pwr_domain_on_finish = rpi3_pwr_domain_on_finish, .system_off = rpi3_system_off,
Jan
Works well, Jailhouse is happy on the RPi4 about the newly gained PSCI support.
How should we continue with the patch below?
If you want to go into the lion's den that is gerrit, please feel free to submit this. Shouldn't be too hard for a single patch. Otherwise I need to look into some RPi4 TF-A stuff this week anyway, so could pick up your patch as well.
Cheers, Andre.
Subject: [PATCH] rpi3/4: Add support for offlining CPUs From: Jan Kiszka jan.kiszka@siemens.com
The hooks were populated but the power down left the CPU in limbo-land. What we need to do - until there is a way to actually power off - is to turn off the MMU and enter the spinning loop as if we were cold-booted. This allows the on-call to pick up the CPU again.
Signed-off-by: Jan Kiszka jan.kiszka@siemens.com
plat/rpi/common/rpi3_pm.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/plat/rpi/common/rpi3_pm.c b/plat/rpi/common/rpi3_pm.c index 8c2d070c4..2a6bf076b 100644 --- a/plat/rpi/common/rpi3_pm.c +++ b/plat/rpi/common/rpi3_pm.c @@ -123,6 +123,15 @@ static void rpi3_pwr_domain_off(const psci_power_state_t *target_state) #endif }
+void __dead2 plat_secondary_cold_boot_setup(void);
+static void __dead2 +rpi3_pwr_domain_pwr_down_wfi(const psci_power_state_t *target_state) +{
- disable_mmu_el3();
- plat_secondary_cold_boot_setup();
+}
/*******************************************************************************
- Platform handler called when a power domain is about to be turned on. The
- mpidr determines the CPU to be turned on.
@@ -224,6 +233,7 @@ static void __dead2 rpi3_system_off(void) static const plat_psci_ops_t plat_rpi3_psci_pm_ops = { .cpu_standby = rpi3_cpu_standby, .pwr_domain_off = rpi3_pwr_domain_off,
- .pwr_domain_pwr_down_wfi = rpi3_pwr_domain_pwr_down_wfi, .pwr_domain_on = rpi3_pwr_domain_on, .pwr_domain_on_finish = rpi3_pwr_domain_on_finish, .system_off = rpi3_system_off,
-- 2.16.4
On 09.12.19 11:07, Andre Przywara wrote:
On Mon, 9 Dec 2019 08:41:47 +0100 Jan Kiszka jan.kiszka@web.de wrote:
Hi,
On 24.11.19 16:08, Jan Kiszka wrote:
On 24.11.19 15:49, Jan Kiszka wrote:
On 24.11.19 15:19, André Przywara wrote:
On 24/11/2019 13:21, Jan Kiszka wrote:
Hi,
I'm trying to get CPU offlining work for Linux on the RPi4. In fact, this is working already with current ATF master and the Raspberry kernel 4.19.85. Not it would be "nice" if onlining worked as well:
Indeed.
[ 94.959294] CPU1: shutdown [ 94.959311] psci: CPU1 killed. [ 106.750960] CPU1: failed to come online [ 106.755425] CPU1: failed in unknown state : 0x0
Yeah, I remember there was an issue with that, but I didn't come around to fix that yet. I would have put this as a limitation in the cover letter, but: gerrit ...
Is this a known limitation? Or am I using the wrong kernel (obviously, it's a downstream one ;) )?
I obviously cannot endorse the usage of downstream kernels ;-), but I think I saw this with mainline as well.
Is mainline working on the RPi4 already? Well, another topic...
Do you get any more messages on the console? I think RPI3_RUNTIME_UART defaults to 0 for the RPI4 (more by accident), which should give you TF-A debug messages on PSCI calls. Maybe you can quickly add some, to see where it breaks?
I've instrumented the psci cpu off/on paths, and the enabling path after disabling never gets us into psci_warmboot_entrypoint() at least.
I might dig out my RPI4, but only later next week, so you might want to beat me on it, if you have your rig set up already.
I this working on the RPi3? I don't see much deviations between both in this regard.
Likely not: rpi3_pwr_domain_off doesn't look to me like it is seriously pulling any plug. And rpi3_pwr_domain_on expect the target to already spin in plat_secondary_cold_boot_setup, in reset state.
Here is a hack to make onlining - apparently - work:
diff --git a/plat/rpi/common/rpi3_pm.c b/plat/rpi/common/rpi3_pm.c index 8c2d070c4..2a6bf076b 100644 --- a/plat/rpi/common/rpi3_pm.c +++ b/plat/rpi/common/rpi3_pm.c @@ -123,6 +123,15 @@ static void rpi3_pwr_domain_off(const psci_power_state_t *target_state) #endif }
+void __dead2 plat_secondary_cold_boot_setup(void);
+static void __dead2 +rpi3_pwr_domain_pwr_down_wfi(const psci_power_state_t *target_state) +{
- disable_mmu_el3();
- plat_secondary_cold_boot_setup();
+}
/*******************************************************************************
- Platform handler called when a power domain is about to be turned on. The
- mpidr determines the CPU to be turned on.
@@ -224,6 +233,7 @@ static void __dead2 rpi3_system_off(void) static const plat_psci_ops_t plat_rpi3_psci_pm_ops = { .cpu_standby = rpi3_cpu_standby, .pwr_domain_off = rpi3_pwr_domain_off,
- .pwr_domain_pwr_down_wfi = rpi3_pwr_domain_pwr_down_wfi, .pwr_domain_on = rpi3_pwr_domain_on, .pwr_domain_on_finish = rpi3_pwr_domain_on_finish, .system_off = rpi3_system_off,
Jan
Works well, Jailhouse is happy on the RPi4 about the newly gained PSCI support.
How should we continue with the patch below?
If you want to go into the lion's den that is gerrit, please feel free to submit this. Shouldn't be too hard for a single patch. Otherwise I need to look into some RPi4 TF-A stuff this week anyway, so could pick up your patch as well.
Ok, I've created
https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/2746 rpi3/4: Add support for offlining CPUs https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/2747 rpi4: Fix documentation of armstub config entry
Hope, that's correct.
Thanks, Jan
tf-a@lists.trustedfirmware.org
-
Andre Przywara -
André Przywara
-
Jan Kiszka