Hi raghu,

Thanks for your guidance! I have never heard of fvp_tb_fw_config.dts before! I will spend much more of time on solving my problems without your help!

Now I can load one OPTEE on at addresses between 0x6000000 and 0x8000000 successfully.

Now I meet two problems, I am trying to be deep into the code to solve the problems by myself. But if you have any idea, I appreciate for your sharing.

Two Problems I met:

1. If I load the OPTEE at address not between 0x6000000 and 0x8000000, like 0x9280000, then there will be error "Invalid SP package magic".

2. I try to load two OPTEEs, the first at address 0x6280000, the second at address 0x7280000, and there is an error "Panic: Could not parse manifest: Property not found."

My effort and the detailed logs are as follows.

==================================My effort and logs for the first problem====================================

1. In the file <path-to-workspace>/optee_os/core/arch/arm/plat-vexpress/conf.mk change "CFG_TZDRAM_START ?=0x06281000" to "CFG_TZDRAM_START ?= 0x09281000"

2. In the file <path-to-workspace>/trusted-firmware-a/fdts/optee_sp_manifest.dts change "load-address = <0x6280000>" to "load-address = <0x9280000>"

3. In the file <path-to-workspace>/trusted-firmware-a/plat/arm/board/fvp/fdts/fvp_tb_fw_config.dts change load-address of op-tee from 0x6280000 to 0x9280000

4. In the file <path-to-workspace>/trusted-firmware-a/plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts change load_address of vm1 from 0x6280000 to 0x9280000

5. Compile OPTEE

CROSS_COMPILE64=aarch64-none-elf- PLATFORM=vexpress PLATFORM_FLAVOR=fvp CFG_ARM_GICV3=y CFG_CORE_SEL2_SPMC=y CFG_ARM64_core=y CFG_USER_TA_TARGETS=ta_arm64 CFG_TEE_CORE_DEBUG=y CFG_TEE_CORE_LOG_LEVEL=4 CFG_TEE_BENCHMARK=n CFG_WITH_STATS=y make

6. Compile ATF

make CROSS_COMPILE=aarch64-none-elf- SPD=spmd CTX_INCLUDE_EL2_REGS=1 PLAT=fvp BL33=../tf-a-tests/build/fvp/debug/tftf.bin DEBUG=1 BL32=../hafnium/out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ARM_ARCH_MINOR=4 SP_LAYOUT_FILE=sp_layout.json ARM_SPMC_MANIFEST_DTS=plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts all fip

7. Run FVP ../fvp/Base_RevC_AEMv8A_pkg/models/Linux64_GCC-6.4/FVP_Base_RevC-2xAEMv8A -C pctl.startup=0.0.0.0 -C cluster0.NUM_CORES=4 -C cluster1.NUM_CORES=4 -C bp.secure_memory=1 -C bp.secureflashloader.fname=trusted-firmware-a/build/fvp/debug/bl1.bin -C bp.flashloader0.fname=trusted-firmware-a/build/fvp/debug/fip.bin -C cluster0.has_arm_v8-4=1 -C cluster1.has_arm_v8-4=1 -C cache_state_modelled=0 -C bp.pl011_uart0.out_file=fvp-uart0.log -C bp.pl011_uart1.out_file=fvp-uart1.log -C bp.vis.disable_visualisation=1

Logs from fvp-uart0.log (fvp-uart1.log is empty)

NOTICE: Booting Trusted Firmware NOTICE: BL1: v2.4(debug):v2.4-79-gb4b23c780-dirty NOTICE: BL1: Built : 15:36:13, Feb 26 2021 INFO: BL1: RAM 0x4035000 - 0x403c000 INFO: Loading image id=31 at address 0x4001010 INFO: Image id=31 loaded: 0x4001010 - 0x4001234 INFO: FCONF: Config file with image ID:31 loaded at address = 0x4001010 INFO: Loading image id=24 at address 0x4001300 INFO: Image id=24 loaded: 0x4001300 - 0x4001466 INFO: FCONF: Config file with image ID:24 loaded at address = 0x4001300 INFO: BL1: Loading BL2 INFO: Loading image id=1 at address 0x4022000 INFO: Image id=1 loaded: 0x4022000 - 0x402bb49 NOTICE: BL1: Booting BL2 INFO: Entry point address = 0x4022000 INFO: SPSR = 0x3c5 INFO: FCONF: Reading FW_CONFIG firmware configuration file from: 0x4001010 INFO: FCONF: Reading firmware configuration information for: dyn_cfg INFO: FCONF: Reading TB_FW firmware configuration file from: 0x4001300 INFO: FCONF: Reading firmware configuration information for: arm_sp NOTICE: BL2: v2.4(debug):v2.4-79-gb4b23c780-dirty NOTICE: BL2: Built : 15:36:16, Feb 26 2021 INFO: BL2: Doing platform setup INFO: Configuring TrustZone Controller INFO: Total 4 regions set. INFO: BL2: Loading image id 3 INFO: Loading image id=3 at address 0x4003000 INFO: Image id=3 loaded: 0x4003000 - 0x401722c INFO: BL2: Loading image id 23 INFO: Loading image id=23 at address 0x82000000 INFO: Image id=23 loaded: 0x82000000 - 0x82002348 INFO: BL2: Loading image id 25 INFO: Loading image id=25 at address 0x4001300 INFO: Image id=25 loaded: 0x4001300 - 0x4001348 INFO: BL2: Loading image id 4 INFO: Loading image id=4 at address 0x6000000 INFO: Image id=4 loaded: 0x6000000 - 0x601e250 INFO: BL2: Skip loading image id 21 INFO: BL2: Skip loading image id 22 INFO: BL2: Loading image id 26 INFO: Loading image id=26 at address 0x4001500 INFO: Image id=26 loaded: 0x4001500 - 0x4001acf INFO: BL2: Loading image id 5 INFO: Loading image id=5 at address 0x88000000 INFO: Image id=5 loaded: 0x88000000 - 0x8800b2b0 INFO: BL2: Loading image id 34 INFO: Loading image id=34 at address 0x9280000 INFO: Image id=34 loaded: 0x9280000 - 0x92e2050 NOTICE: BL1: Booting BL31 INFO: Entry point address = 0x4003000 INFO: SPSR = 0x3cd INFO: FCONF: Reading HW_CONFIG firmware configuration file from: 0x82000000 INFO: FCONF: Reading firmware configuration information for: cpu_timer INFO: FCONF: Reading firmware configuration information for: uart_config INFO: FCONF: Reading firmware configuration information for: topology INFO: FCONF: Reading firmware configuration information for: gicv3_config NOTICE: BL31: v2.4(debug):v2.4-79-gb4b23c780-dirty NOTICE: BL31: Built : 15:36:21, Feb 26 2021 INFO: GICv3 with legacy support detected. INFO: ARM GICv3 driver initialized in EL3 INFO: BL31: Initializing runtime services INFO: SPM Core setup done. INFO: BL31: Initializing BL32 NOTICE: Initialising hafnium INFO: text: 0x6000000 - 0x601a000 INFO: rodata: 0x601a000 - 0x601e000 INFO: data: 0x601e000 - 0x60a0000 INFO: Supported bits in physical address: 40 INFO: Stage 2 has 3 page table levels with 2 pages at the root. INFO: SPM: 0x84000069 0x0 0x0 0x0 0x0 0x0 0x0 0x0 INFO: Memory range: 0x6000000 - 0xaffffff ERROR: Invalid SP package magic. the invalid magic number is 0xe7ff0010 Panic: Could not parse manifest: Hypervisor manifest entry not compatible with Hafnium.

================================My effort and logs for the second problem===================================

This part is sent by another email because of the limit of body length

=====================================================================================================

Now I am trying to be deep into the code to solve the problems by myself. But if anyone have any idea, I appreciate for your sharing.

Sincerely, Yinhua ________________________________ From: raghu.ncstate@icloud.com raghu.ncstate@icloud.com Sent: Friday, February 26, 2021 5:35 To: yinhua_chen@outlook.com yinhua_chen@outlook.com; hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org Subject: RE: [Hafnium] Trying to load two OPTEEs on hafnium

From the log it looks like there was nothing loaded at 0x8001000. Possible that you are missing the second entry for OPTEE in the SPMC manifest(fvp_spmc_optee_sp_manifest.dts). Might want to use the sptool to check if both your optee images were packaged. If not, you likely have your answer. If it is present, might want to make sure that fvp_tb_fw_config.dts has an entry for the second OPTEE partition to be loaded by BL2 at 0x8001000.

For the partition to run, both OPTEE images have to be loaded by BL2, and both OPTEE images have processed by hafnium during initialization(when it processes the SPMC manifest).

Thanks Raghu

-----Original Message----- From: Hafnium hafnium-bounces@lists.trustedfirmware.org On Behalf Of yinhua_chen--- via Hafnium Sent: Wednesday, February 24, 2021 7:23 PM To: hafnium@lists.trustedfirmware.org Subject: [Hafnium] Trying to load two OPTEEs on hafnium

Hi all,

I am trying to load two OPTEEs on hafnium.

Thanks to the help from Hafnium open source community, now I can load one OPTEE on hafnium. The guidance I am following is attached at the end.

In the guidance, the OPTEE is loaded at address 0x06280000.

I think the first step to load two OPTEEs is to load an OPTEE at an address other than 0x06280000. I tried but failed.

If there is anyone who can give me some advice, I would be very grateful.

=====================The following is the effort I made===============

# NOTE: make sure gcc-arm-9.2-2019.12-x86_64-aarch64-none-elf/bin is in PATH

1. Loading one OPTEE at address 0x06280000 successfully by following the guidance attached below.

2. In the file <path-to-workspace>/optee_os/core/arch/arm/plat-vexpress/conf.mk change "CFG_TZDRAM_START ?=0x06281000" to "CFG_TZDRAM_START ?= 0x08002000"

3. In the file <path-to-workspace>//trusted-firmware-a/fdts/optee_sp_manifest.dts change "load-address = <0x6280000>" to "load-address = <0x8001000>"

4. Compile OPTEE CROSS_COMPILE64=aarch64-none-elf- PLATFORM=vexpress PLATFORM_FLAVOR=fvp CFG_ARM_GICV3=y CFG_CORE_SEL2_SPMC=y CFG_ARM64_core=y CFG_USER_TA_TARGETS=ta_arm64 CFG_TEE_CORE_DEBUG=y CFG_TEE_CORE_LOG_LEVEL=4 CFG_TEE_BENCHMARK=n CFG_WITH_STATS=y make

5. Compile ATF make CROSS_COMPILE=aarch64-none-elf- SPD=spmd CTX_INCLUDE_EL2_REGS=1 PLAT=fvp BL33=../tf-a-tests/build/fvp/debug/tftf.bin DEBUG=1 BL32=../hafnium/out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ARM_ARCH_MINOR=4 SP_LAYOUT_FILE=sp_layout.json ARM_SPMC_MANIFEST_DTS=plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts all fip

6. Run FVP ../fvp/Base_RevC_AEMv8A_pkg/models/Linux64_GCC-6.4/FVP_Base_RevC-2xAEMv8A -C pctl.startup=0.0.0.0 -C cluster0.NUM_CORES=4 -C cluster1.NUM_CORES=4 -C bp.secure_memory=1 -C bp.secureflashloader.fname=trusted-firmware-a/build/fvp/debug/bl1.bin -C bp.flashloader0.fname=trusted-firmware-a/build/fvp/debug/fip.bin -C cluster0.has_arm_v8-4=1 -C cluster1.has_arm_v8-4=1 -C cache_state_modelled=0 -C bp.pl011_uart0.out_file=fvp-uart0.log -C bp.pl011_uart1.out_file=fvp-uart1.log -C bp.vis.disable_visualisation=1

7. The error message from fvp-uart0.log (fvp-uart1.log is empty)

NOTICE: Booting Trusted Firmware NOTICE: BL1: v2.4(debug):v2.4-79-gb4b23c780-dirty NOTICE: BL1: Built : 09:39:24, Feb 25 2021 INFO: BL1: RAM 0x4035000 - 0x403c000 INFO: Loading image id=31 at address 0x4001010 INFO: Image id=31 loaded: 0x4001010 - 0x4001234 INFO: FCONF: Config file with image ID:31 loaded at address = 0x4001010 INFO: Loading image id=24 at address 0x4001300 INFO: Image id=24 loaded: 0x4001300 - 0x4001466 INFO: FCONF: Config file with image ID:24 loaded at address = 0x4001300 INFO: BL1: Loading BL2 INFO: Loading image id=1 at address 0x4022000 INFO: Image id=1 loaded: 0x4022000 - 0x402bb49 NOTICE: BL1: Booting BL2 INFO: Entry point address = 0x4022000 INFO: SPSR = 0x3c5 INFO: FCONF: Reading FW_CONFIG firmware configuration file from: 0x4001010 INFO: FCONF: Reading firmware configuration information for: dyn_cfg INFO: FCONF: Reading TB_FW firmware configuration file from: 0x4001300 INFO: FCONF: Reading firmware configuration information for: arm_sp NOTICE: BL2: v2.4(debug):v2.4-79-gb4b23c780-dirty NOTICE: BL2: Built : 09:39:26, Feb 25 2021 INFO: BL2: Doing platform setup INFO: Configuring TrustZone Controller INFO: Total 4 regions set. INFO: BL2: Loading image id 3 INFO: Loading image id=3 at address 0x4003000 INFO: Image id=3 loaded: 0x4003000 - 0x401722c INFO: BL2: Loading image id 23 INFO: Loading image id=23 at address 0x82000000 INFO: Image id=23 loaded: 0x82000000 - 0x82002348 INFO: BL2: Loading image id 25 INFO: Loading image id=25 at address 0x4001300 INFO: Image id=25 loaded: 0x4001300 - 0x4001348 INFO: BL2: Loading image id 4 INFO: Loading image id=4 at address 0x6000000 INFO: Image id=4 loaded: 0x6000000 - 0x601e250 INFO: BL2: Skip loading image id 21 INFO: BL2: Skip loading image id 22 INFO: BL2: Loading image id 26 INFO: Loading image id=26 at address 0x4001500 INFO: Image id=26 loaded: 0x4001500 - 0x4001acf INFO: BL2: Loading image id 5 INFO: Loading image id=5 at address 0x88000000 INFO: Image id=5 loaded: 0x88000000 - 0x8800b2b0 INFO: BL2: Loading image id 34 INFO: Loading image id=34 at address 0x6280000 INFO: Image id=34 loaded: 0x6280000 - 0x62e2050 NOTICE: BL1: Booting BL31 INFO: Entry point address = 0x4003000 INFO: SPSR = 0x3cd INFO: FCONF: Reading HW_CONFIG firmware configuration file from: 0x82000000 INFO: FCONF: Reading firmware configuration information for: cpu_timer INFO: FCONF: Reading firmware configuration information for: uart_config INFO: FCONF: Reading firmware configuration information for: topology INFO: FCONF: Reading firmware configuration information for: gicv3_config NOTICE: BL31: v2.4(debug):v2.4-79-gb4b23c780-dirty NOTICE: BL31: Built : 09:39:31, Feb 25 2021 INFO: GICv3 with legacy support detected. INFO: ARM GICv3 driver initialized in EL3 INFO: BL31: Initializing runtime services INFO: SPM Core setup done. INFO: BL31: Initializing BL32 NOTICE: Initialising hafnium INFO: text: 0x6000000 - 0x601a000 INFO: rodata: 0x601a000 - 0x601e000 INFO: data: 0x601e000 - 0x60a0000 INFO: Supported bits in physical address: 40 INFO: Stage 2 has 3 page table levels with 2 pages at the root. INFO: SPM: 0x84000069 0x0 0x0 0x0 0x0 0x0 0x0 0x0 INFO: Memory range: 0x6000000 - 0x7ffffff INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x0 0x6001340 0x601e000 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10300 0x6001340 0x601e0a8 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10200 0x6001340 0x601e090 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10100 0x6001340 0x601e078 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10000 0x6001340 0x601e060 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x300 0x6001340 0x601e048 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x200 0x6001340 0x601e030 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x100 0x6001340 0x601e018 0x0 INFO: In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself) INFO: Loaded primary VM with 8 vCPUs, entry at 0x8001000. INFO: Hafnium initialisation completed WARNING: Stage-2 page fault: pc=0x8002000, vmid=32769, vcpu=0, vaddr=0x8002000, ipaddr=0x0, mode=0x4 NOTICE: Injecting Instruction Abort exception into VM32769. WARNING: Stage-2 page fault: pc=0x200, vmid=32769, vcpu=0, vaddr=0x200, ipaddr=0x200, mode=0x4 NOTICE: Injecting Instruction Abort exception into VM32769. WARNING: Stage-2 page fault: pc=0x200, vmid=32769, vcpu=0, vaddr=0x200, ipaddr=0x200, mode=0x4 NOTICE: Injecting Instruction Abort exception into VM32769. WARNING: Stage-2 page fault: pc=0x200, vmid=32769, vcpu=0, vaddr=0x200, ipaddr=0x200, mode=0x4 NOTICE: Injecting Instruction Abort exception into VM32769. . . . (infinite loop . . .)

============The guidance from Oliver Deprez to load one OPTEE============

# NOTE: make sure gcc-arm-9.2-2019.12-x86_64-aarch64-none-elf/bin is in PATH

mkdir workspace; cd workspace

# Hafnium # https://review.trustedfirmware.org/plugins/gitiles/hafnium/hafnium/+/HEAD/do...

git clone https://git.trustedfirmware.org/hafnium/hafnium.git; cd hafnium git fetch "https://review.trustedfirmware.org/hafnium/hafnium" refs/changes/11/6011/16 && git checkout FETCH_HEAD git checkout -b spmc git submodule update --init make PROJECT=reference cd ..

# OPTEE # https://optee.readthedocs.io/en/latest/building/prerequisites.html#prerequis...

git clone https://git.trustedfirmware.org/OP-TEE/optee_os.git -b ffa_rel_proto; cd optee_os

# Adjust TEE start address perl -pi -e 's/CFG_TZDRAM_START ?= 0x06280000/CFG_TZDRAM_START ?= 0x06281000/' core/arch/arm/plat-vexpress/conf.mk

CROSS_COMPILE64=aarch64-none-elf- PLATFORM=vexpress PLATFORM_FLAVOR=fvp CFG_ARM_GICV3=y CFG_CORE_SEL2_SPMC=y CFG_ARM64_core=y CFG_USER_TA_TARGETS=ta_arm64 CFG_TEE_CORE_DEBUG=y CFG_TEE_CORE_LOG_LEVEL=4 CFG_TEE_BENCHMARK=n CFG_WITH_STATS=y make cd ..

# TF-A-tests

git clone https://git.trustedfirmware.org/TF-A/tf-a-tests.git; cd tf-a-tests make CROSS_COMPILE=aarch64-none-elf- PLAT=fvp DEBUG=1 TESTS=spm -j4 cd ..

# TF-A # git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git; cd trusted-firmware-a git checkout b4b23c780c0763ce01ac21de1a0e79287677405f -b spmc

#{ # "op-tee" : { # "image": "../optee_os/out/arm-plat-vexpress/core/tee-pager_v2.bin", # "pm": "fdts/optee_sp_manifest.dts" # } #} echo \ "{ \ "op-tee" : { \ "image": "../optee_os/out/arm-plat-vexpress/core/tee-pager_v2.bin", \ "pm": "fdts/optee_sp_manifest.dts" \ } }" > sp_layout.json

make CROSS_COMPILE=aarch64-none-elf- SPD=spmd CTX_INCLUDE_EL2_REGS=1 PLAT=fvp BL33=../tf-a-tests/build/fvp/debug/tftf.bin DEBUG=1 BL32=../hafnium/out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ARM_ARCH_MINOR=4 SP_LAYOUT_FILE=sp_layout.json ARM_SPMC_MANIFEST_DTS=plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts all fip cd ..

# Run FVP

../fvp/Base_RevC_AEMv8A_pkg/models/Linux64_GCC-6.4/FVP_Base_RevC-2xAEMv8A -C pctl.startup=0.0.0.0 -C cluster0.NUM_CORES=4 -C cluster1.NUM_CORES=4 -C bp.secure_memory=1 -C bp.secureflashloader.fname=trusted-firmware-a/build/fvp/debug/bl1.bin -C bp.flashloader0.fname=trusted-firmware-a/build/fvp/debug/fip.bin -C cluster0.has_arm_v8-4=1 -C cluster1.has_arm_v8-4=1 -C cache_state_modelled=0 -C bp.pl011_uart0.out_file=fvp-uart0.log -C bp.pl011_uart1.out_file=fvp-uart1.log -C bp.vis.disable_visualisation=1

=============================

I appreciate for any possible help!

Thanks

Yinhua -- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium