Re: [Hafnium] Trying to load two OPTEEs on hafnium - Hafnium

28 Feb 2021


      Hi Oilvier,
Want some clarification on this statement: " A configuration with multiple OP-TEE instances is valid although (to my knowledge) this probably lacks the appropriate OP-TEE support and has never been tested."
Not sure I understand why OPTEE needs support for multiple OPTEE vm's to run on SPMC. If OPTEE needs to be aware that there are other OPTEE partitions, then we are likely doing something wrong with the abstractions. If SPMC supports booting multiple SP's, it should not matter how many other partitions are there or booted unless there is a dependency(which in this case should not exist).
Having said that, I assume you mainly wanted to point out that this has never been tested? If not, what support would OPTEE require to boot as multiple instances on an SPMC beyond what Yinhua is doing? What is being done *should* be supported with the latest code base.
Thanks
Raghu
-----Original Message-----
From: Hafnium hafnium-bounces@lists.trustedfirmware.org On Behalf Of Olivier Deprez via Hafnium
Sent: Sunday, February 28, 2021 7:02 AM
To: hafnium@lists.trustedfirmware.org; 胤桦 陈 yinhua_chen@outlook.com
Subject: Re: [Hafnium] Trying to load two OPTEEs on hafnium
Hi,
For the first problem: on the FVP platform, Hafnium (SPMC) and Secure Partitions are meant to reside in the "32MB Trusted DRAM" region (0x6000000-0x7ffffff). The memory node in SPMC manifest describes the (collection of) system memory range(s) that the SPMC and SPs are allowed to use. This region is always marked secure on this platform. Using other regions is possible provided they're marked as secure (e.g. by the TZC400 controller) and declared in the memory node (list of start addresses and sizes).
For the second problem: the Hafnium code base used in this sample is rather old and the properties specified in the SPMC manifest are different for the first and other SPs. The first SP fixes vCPU count to number of PEs and maps the full system memory. Other SPs must declare the number of vCPUs and SP memory size in the manifest (https://review.trustedfirmware.org/plugins/gitiles/hafnium/hafnium/+/HEAD/do...)
In your sample case you need:
vm2 {
            is_ffa_partition;
            debug_name = "op-tee2";
            load_address = <0x7280000>;
            smc_whitelist = <0xce000000>;
            vcpu_count = <8>;
            mem_size = <1048576>;
      };
Using recent Hafnium master, all SPs including the first one declare a vCPU count and SP size in the manifest (http://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/tree/plat/arm/boa...)
Note you'd also want to use different UUIDs for both partitions.
You can get more details on secure partitions loading from https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partiti...
On the rationale for this experiment:
Hafnium effectively supports booting multiple partitions.
OP-TEE has a build option to run as a single SP on top of the SPMC.
A configuration with multiple OP-TEE instances is valid although (to my knowledge) this probably lacks the appropriate OP-TEE support and has never been tested.
It may be good understanding your use case better, whether you wish to strictly demo the Hafnium multiple partitions support, or you really need to bring up multiple OP-TEE instances.
In other words, would it be enough to host services in bare metal SPs?
Would a single OP-TEE partition (redirecting FF-A messages to Trusted Applications implementing secure services) also be acceptable?
Regards,
Olivier.
________________________________________
From: Hafnium hafnium-bounces@lists.trustedfirmware.org on behalf of 胤桦 陈 via Hafnium hafnium@lists.trustedfirmware.org
Sent: 27 February 2021 04:11
To: raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org
Subject: Re: [Hafnium] Trying to load two OPTEEs on hafnium
Hi raghu,
For the second problem, the memory range is set by myself. The corresponding property is
memory@6000000 {
         device_type = "memory";
         reg = <0x0 0x6000000 0x5000000>; /* Trusted DRAM */ };
in the file fvp_spmc_optee_sp_manifest.dts.
The original setting is reg = <0x0 0x6000000 0x2000000>. Since I find that OPTEE only works fine when loaded at address between 0x6000000 and 0x2000000, I guess the reason is that 0x9280000 is not in the "Trusted DRAM", so I try to reset it to reg = <0x0 0x6000000 0x5000000>. However, it seems not to be the solution.
And, I try to load OPTEE at 0x7281000 and it works fine, no matter with  reg = <0x0 0x6000000 0x2000000> or reg = <0x0 0x6000000 0x5000000>.
Appreciate for your help.
Sincerely,
Yinhua
________________________________
From: raghu.ncstate@icloud.com raghu.ncstate@icloud.com
Sent: Saturday, February 27, 2021 2:40
To: 'Manish Pandey2' Manish.Pandey2@arm.com; hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org; '胤桦 陈' yinhua_chen@outlook.com
Subject: RE: [Hafnium] Trying to load two OPTEEs on hafnium
Yep, I was just about to reply. First problem is likely because of loading at 0x9280000.
For the second problem, I see “INFO: Memory range:  0x6000000 - 0xaffffff”. On a good run, it should show the memory range as 0x6000000 – 0x7FFFFFF(see the long in your original email). So either your spmc manifest is incorrect or there is something overwriting the memory location of manifest, which may or may not be related to a problem, but needs to be addressed anyway
Would it be possible to run a single OPTEE but loaded at 0x7281000 and see if it works fine? That would also give another data point to diagnose the issue.
From: Manish Pandey2 Manish.Pandey2@arm.com
Sent: Friday, February 26, 2021 10:14 AM
To: raghu.ncstate@icloud.com; hafnium@lists.trustedfirmware.org; 胤桦 陈 yinhua_chen@outlook.com
Subject: Re: [Hafnium] Trying to load two OPTEEs on hafnium
Hi Yinhua,
I reproduced the first problem at my end, please find my analysis below
If you look at FVP memory map https://developer.arm.com/documentation/100966/1111-00/Arm-Neoverse-referenc... address 0x9280000 is part of NOR flash which is read-only.
Ideally this should have been mapped as RO in BL2's page table, but it is currently mapped as RW at plat/arm/board/fvp/fvp_common.c +83 (mostly a bug, will investigate it)
I did a quick experiment and changed this mapping to RO and system stuck while loading SP
INFO:    Loading image id=34 at address 0x9280000
INFO:    Image id=34 loaded: 0x9280000 - 0x92e2050
What's happening is, BL2 writes at RO area and thinks that write has been successful and when hafnium tries to match the SP MAGIC it fails.
Will try to investigate your second problem on Monday and let you know.
Thanks
Manish
________________________________
From: Hafnium <hafnium-bounces@lists.trustedfirmware.orgmailto:hafnium-bounces@lists.trustedfirmware.org> on behalf of 胤桦 陈 via Hafnium <hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org>
Sent: 26 February 2021 08:59
To: raghu.ncstate@icloud.commailto:raghu.ncstate@icloud.com <raghu.ncstate@icloud.commailto:raghu.ncstate@icloud.com>; hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org <hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org>
Subject: Re: [Hafnium] Trying to load two OPTEEs on hafnium
This is the part "My effort and logs for the second problem"
================================My effort and logs for the second problem===================================
1. copy the optee_os directory:
                cp -r <path-to-workspace>/optee_os    <path-to-workspace>/optee_os2
              mv <path-to-workspace>/optee_os <path-to-workspace>/optee_os1
2. Set both OPTEE's conf.mk file
              In the file <path-to-workspace>/optee_os1/core/arch/arm/plat-vexpress/conf.mk
                             set CFG_TZDRAM_START ?=0x06281000
              In the file <path-to-workspace>/optee_os2/core/arch/arm/plat-vexpress/conf.mk
                             set CFG_TZDRAM_START ?=0x07281000
3. Set both OPTEE's optee_sp_manifest.dts
             cp <path-to-workspace>/trusted-firmware-a/fdts/optee_sp_manifest.dts   <path-to-workspace>/trusted-firmware-a/fdts/optee2_sp_manifest.dts
             mv <path-to-workspace>/trusted-firmware-a/fdts/optee_sp_manifest.dts   <path-to-workspace>/trusted-firmware-a/fdts/optee1_sp_manifest.dts
             In the file <path-to-workspace>/trusted-firmware-a/fdts/optee1_sp_manifest.dts
                       set  id = <1>;  load-address = <0x6280000>;
             In the file <path-to-workspace>/trusted-firmware-a/fdts/optee2_sp_manifest.dts
                       set  id = <2>;  load-address = <0x7280000>;
4. In the file <path-to-workspace>/trusted-firmware-a/plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts
             Set "hypervisor" node like following
hypervisor {
      compatible = "hafnium,hafnium";
vm1 {
            is_ffa_partition;
            debug_name = "op-tee1";
            load_address = <0x6280000>;
            smc_whitelist = <0xbe000000>;
      };
vm2 {
            is_ffa_partition;
            debug_name = "op-tee2";
            load_address = <0x7280000>;
            smc_whitelist = <0xce000000>;
      };
};
5. In the file <path-to-workspace>/trusted-firmware-a/plat/arm/board/fvp/fdts/fvp_tb_fw_config.dts
             Set "secure-partitions" node like following
secure-partitions {
      compatible = "arm,sp";
#ifdef OPTEE_SP_FW_CONFIG
      op-tee1 {
            uuid = <0x486178e0 0xe7f811e3 0xbc5e0002 0xa5d5c51b>;
            load-address = <0x6280000>;
      };
op-tee2 {
            uuid = <0x486178e0 0xe7f811e3 0xbc5e0002 0xa5d5c51b>;
            load-address = <0x7280000>;
      };
#else
.
. (cactus)
.
#endif
};
6. Compile OPTEE1 and OPTEE2
7. Compile ATF
8. Run FVP
Logs from fvp-uart0.log (fvp-uart1.log is empty)
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL1: Built : 15:36:13, Feb 26 2021
INFO:    BL1: RAM 0x4035000 - 0x403c000
INFO:    Loading image id=31 at address 0x4001010
INFO:    Image id=31 loaded: 0x4001010 - 0x4001234
INFO:    FCONF: Config file with image ID:31 loaded at address = 0x4001010
INFO:    Loading image id=24 at address 0x4001300
INFO:    Image id=24 loaded: 0x4001300 - 0x40014a2
INFO:    FCONF: Config file with image ID:24 loaded at address = 0x4001300
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0x4022000
INFO:    Image id=1 loaded: 0x4022000 - 0x402bb49
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0x4022000
INFO:    SPSR = 0x3c5
INFO:    FCONF: Reading FW_CONFIG firmware configuration file from: 0x4001010
INFO:    FCONF: Reading firmware configuration information for: dyn_cfg
INFO:    FCONF: Reading TB_FW firmware configuration file from: 0x4001300
INFO:    FCONF: Reading firmware configuration information for: arm_sp
NOTICE:  BL2: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL2: Built : 15:36:16, Feb 26 2021
INFO:    BL2: Doing platform setup
INFO:    Configuring TrustZone Controller
INFO:    Total 4 regions set.
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0x4003000
INFO:    Image id=3 loaded: 0x4003000 - 0x401722c
INFO:    BL2: Loading image id 23
INFO:    Loading image id=23 at address 0x82000000
INFO:    Image id=23 loaded: 0x82000000 - 0x82002348
INFO:    BL2: Loading image id 25
INFO:    Loading image id=25 at address 0x4001300
INFO:    Image id=25 loaded: 0x4001300 - 0x4001348
INFO:    BL2: Loading image id 4
INFO:    Loading image id=4 at address 0x6000000
INFO:    Image id=4 loaded: 0x6000000 - 0x601e250
INFO:    BL2: Skip loading image id 21
INFO:    BL2: Skip loading image id 22
INFO:    BL2: Loading image id 26
INFO:    Loading image id=26 at address 0x4001500
INFO:    Image id=26 loaded: 0x4001500 - 0x4001b1b
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x88000000
INFO:    Image id=5 loaded: 0x88000000 - 0x8800b2b0
INFO:    BL2: Loading image id 34
INFO:    Loading image id=34 at address 0x6280000
INFO:    Image id=34 loaded: 0x6280000 - 0x62e2050
INFO:    BL2: Loading image id 35
INFO:    Loading image id=35 at address 0x7280000
INFO:    Image id=35 loaded: 0x7280000 - 0x72e2050
NOTICE:  BL1: Booting BL31
INFO:    Entry point address = 0x4003000
INFO:    SPSR = 0x3cd
INFO:    FCONF: Reading HW_CONFIG firmware configuration file from: 0x82000000
INFO:    FCONF: Reading firmware configuration information for: cpu_timer
INFO:    FCONF: Reading firmware configuration information for: uart_config
INFO:    FCONF: Reading firmware configuration information for: topology
INFO:    FCONF: Reading firmware configuration information for: gicv3_config
NOTICE:  BL31: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL31: Built : 15:36:21, Feb 26 2021
INFO:    GICv3 with legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    BL31: Initializing runtime services
INFO:    SPM Core setup done.
INFO:    BL31: Initializing BL32
NOTICE: Initialising hafnium
INFO: text: 0x6000000 - 0x601a000
INFO: rodata: 0x601a000 - 0x601e000
INFO: data: 0x601e000 - 0x60a0000
INFO: Supported bits in physical address: 40
INFO: Stage 2 has 3 page table levels with 2 pages at the root.
INFO:    SPM: 0x84000069 0x0 0x0 0x0 0x0 0x0 0x0 0x0
INFO: Memory range:  0x6000000 - 0xaffffff
INFO: Valid SP package magic. the valid magic number is 0x474b5053
Panic: Could not parse manifest: Property not found.
======================================================================
Now I am trying to be deep into the code to solve the problems by myself. But if anyone have any idea, I appreciate for your sharing.
Sincerely,
Yinhua
________________________________
From: Hafnium <hafnium-bounces@lists.trustedfirmware.orgmailto:hafnium-bounces@lists.trustedfirmware.org> on behalf of 胤桦 陈 via Hafnium <hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org>
Sent: Friday, February 26, 2021 16:48
To: raghu.ncstate@icloud.commailto:raghu.ncstate@icloud.com <raghu.ncstate@icloud.commailto:raghu.ncstate@icloud.com>; hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org <hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org>
Subject: Re: [Hafnium] Trying to load two OPTEEs on hafnium
Hi raghu,
Thanks for your guidance! I have never heard of fvp_tb_fw_config.dts before! I will spend much more of time on solving my problems without your help!
Now I can load one OPTEE on at addresses between 0x6000000 and 0x8000000 successfully.
Now I meet two problems, I am trying to be deep into the code to solve the problems by myself. But if you have any idea, I appreciate for your sharing.
Two Problems I met:
1. If I load the OPTEE at address not between 0x6000000 and 0x8000000, like 0x9280000, then there will be error "Invalid SP package magic".
2. I try to load two OPTEEs, the first at address 0x6280000, the second at address 0x7280000, and there is an error "Panic: Could not parse manifest: Property not found."
My effort and the detailed logs are as follows.
==================================My effort and logs for the first problem====================================
1. In the file <path-to-workspace>/optee_os/core/arch/arm/plat-vexpress/conf.mk
              change "CFG_TZDRAM_START ?=0x06281000" to "CFG_TZDRAM_START ?= 0x09281000"
2. In the file <path-to-workspace>/trusted-firmware-a/fdts/optee_sp_manifest.dts
             change  "load-address = <0x6280000>" to "load-address = <0x9280000>"
3. In the file <path-to-workspace>/trusted-firmware-a/plat/arm/board/fvp/fdts/fvp_tb_fw_config.dts
             change  load-address of op-tee from 0x6280000 to 0x9280000
4. In the file <path-to-workspace>/trusted-firmware-a/plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts
             change load_address of vm1 from 0x6280000 to 0x9280000
5. Compile OPTEE
CROSS_COMPILE64=aarch64-none-elf- PLATFORM=vexpress PLATFORM_FLAVOR=fvp CFG_ARM_GICV3=y CFG_CORE_SEL2_SPMC=y CFG_ARM64_core=y CFG_USER_TA_TARGETS=ta_arm64 CFG_TEE_CORE_DEBUG=y CFG_TEE_CORE_LOG_LEVEL=4 CFG_TEE_BENCHMARK=n CFG_WITH_STATS=y make
6. Compile ATF
make CROSS_COMPILE=aarch64-none-elf- SPD=spmd CTX_INCLUDE_EL2_REGS=1 PLAT=fvp BL33=../tf-a-tests/build/fvp/debug/tftf.bin DEBUG=1 BL32=../hafnium/out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ARM_ARCH_MINOR=4 SP_LAYOUT_FILE=sp_layout.json ARM_SPMC_MANIFEST_DTS=plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts all fip
7. Run FVP
../fvp/Base_RevC_AEMv8A_pkg/models/Linux64_GCC-6.4/FVP_Base_RevC-2xAEMv8A -C pctl.startup=0.0.0.0 -C cluster0.NUM_CORES=4 -C cluster1.NUM_CORES=4 -C bp.secure_memory=1 -C bp.secureflashloader.fname=trusted-firmware-a/build/fvp/debug/bl1.bin -C bp.flashloader0.fname=trusted-firmware-a/build/fvp/debug/fip.bin -C cluster0.has_arm_v8-4=1 -C cluster1.has_arm_v8-4=1 -C cache_state_modelled=0 -C bp.pl011_uart0.out_file=fvp-uart0.log -C bp.pl011_uart1.out_file=fvp-uart1.log -C bp.vis.disable_visualisation=1
Logs from fvp-uart0.log (fvp-uart1.log is empty)
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL1: Built : 15:36:13, Feb 26 2021
INFO:    BL1: RAM 0x4035000 - 0x403c000
INFO:    Loading image id=31 at address 0x4001010
INFO:    Image id=31 loaded: 0x4001010 - 0x4001234
INFO:    FCONF: Config file with image ID:31 loaded at address = 0x4001010
INFO:    Loading image id=24 at address 0x4001300
INFO:    Image id=24 loaded: 0x4001300 - 0x4001466
INFO:    FCONF: Config file with image ID:24 loaded at address = 0x4001300
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0x4022000
INFO:    Image id=1 loaded: 0x4022000 - 0x402bb49
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0x4022000
INFO:    SPSR = 0x3c5
INFO:    FCONF: Reading FW_CONFIG firmware configuration file from: 0x4001010
INFO:    FCONF: Reading firmware configuration information for: dyn_cfg
INFO:    FCONF: Reading TB_FW firmware configuration file from: 0x4001300
INFO:    FCONF: Reading firmware configuration information for: arm_sp
NOTICE:  BL2: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL2: Built : 15:36:16, Feb 26 2021
INFO:    BL2: Doing platform setup
INFO:    Configuring TrustZone Controller
INFO:    Total 4 regions set.
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0x4003000
INFO:    Image id=3 loaded: 0x4003000 - 0x401722c
INFO:    BL2: Loading image id 23
INFO:    Loading image id=23 at address 0x82000000
INFO:    Image id=23 loaded: 0x82000000 - 0x82002348
INFO:    BL2: Loading image id 25
INFO:    Loading image id=25 at address 0x4001300
INFO:    Image id=25 loaded: 0x4001300 - 0x4001348
INFO:    BL2: Loading image id 4
INFO:    Loading image id=4 at address 0x6000000
INFO:    Image id=4 loaded: 0x6000000 - 0x601e250
INFO:    BL2: Skip loading image id 21
INFO:    BL2: Skip loading image id 22
INFO:    BL2: Loading image id 26
INFO:    Loading image id=26 at address 0x4001500
INFO:    Image id=26 loaded: 0x4001500 - 0x4001acf
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x88000000
INFO:    Image id=5 loaded: 0x88000000 - 0x8800b2b0
INFO:    BL2: Loading image id 34
INFO:    Loading image id=34 at address 0x9280000
INFO:    Image id=34 loaded: 0x9280000 - 0x92e2050
NOTICE:  BL1: Booting BL31
INFO:    Entry point address = 0x4003000
INFO:    SPSR = 0x3cd
INFO:    FCONF: Reading HW_CONFIG firmware configuration file from: 0x82000000
INFO:    FCONF: Reading firmware configuration information for: cpu_timer
INFO:    FCONF: Reading firmware configuration information for: uart_config
INFO:    FCONF: Reading firmware configuration information for: topology
INFO:    FCONF: Reading firmware configuration information for: gicv3_config
NOTICE:  BL31: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL31: Built : 15:36:21, Feb 26 2021
INFO:    GICv3 with legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    BL31: Initializing runtime services
INFO:    SPM Core setup done.
INFO:    BL31: Initializing BL32
NOTICE: Initialising hafnium
INFO: text: 0x6000000 - 0x601a000
INFO: rodata: 0x601a000 - 0x601e000
INFO: data: 0x601e000 - 0x60a0000
INFO: Supported bits in physical address: 40
INFO: Stage 2 has 3 page table levels with 2 pages at the root.
INFO:    SPM: 0x84000069 0x0 0x0 0x0 0x0 0x0 0x0 0x0
INFO: Memory range:  0x6000000 - 0xaffffff
ERROR: Invalid SP package magic. the invalid magic number is 0xe7ff0010
Panic: Could not parse manifest: Hypervisor manifest entry not compatible with Hafnium.
================================My effort and logs for the second problem===================================
This part is sent by another email because of the limit of body length
=====================================================================================================
Now I am trying to be deep into the code to solve the problems by myself. But if anyone have any idea, I appreciate for your sharing.
Sincerely,
Yinhua
________________________________
From: raghu.ncstate@icloud.commailto:raghu.ncstate@icloud.com <raghu.ncstate@icloud.commailto:raghu.ncstate@icloud.com>
Sent: Friday, February 26, 2021 5:35
To: yinhua_chen@outlook.commailto:yinhua_chen@outlook.com <yinhua_chen@outlook.commailto:yinhua_chen@outlook.com>; hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org <hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org>
Subject: RE: [Hafnium] Trying to load two OPTEEs on hafnium
...
From the log it looks like there was nothing loaded at 0x8001000. Possible that you are missing the second entry for OPTEE in the SPMC manifest(fvp_spmc_optee_sp_manifest.dts). Might want to use the sptool to check if both your optee images were packaged. If not, you likely have your answer. If it is present, might want to make sure that fvp_tb_fw_config.dts has an entry for the second OPTEE partition to be loaded by BL2 at 0x8001000.
For the partition to run,  both OPTEE images have to be loaded by BL2, and both OPTEE images have processed by hafnium during initialization(when it processes the SPMC manifest).
Thanks
Raghu
-----Original Message-----
From: Hafnium <hafnium-bounces@lists.trustedfirmware.orgmailto:hafnium-bounces@lists.trustedfirmware.org> On Behalf Of yinhua_chen--- via Hafnium
Sent: Wednesday, February 24, 2021 7:23 PM
To: hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org
Subject: [Hafnium] Trying to load two OPTEEs on hafnium
Hi all,
I am trying to load two OPTEEs on hafnium.
Thanks to the help from Hafnium open source community, now I can load one OPTEE on hafnium. The guidance I am following is attached at the end.
In the guidance, the OPTEE is loaded at address 0x06280000.
I think the first step to load two OPTEEs is to load an OPTEE at an address other than 0x06280000. I tried but failed.
If there is anyone who can give me some advice, I would be very grateful.
=====================The following is the effort I made===============
# NOTE: make sure gcc-arm-9.2-2019.12-x86_64-aarch64-none-elf/bin is in PATH
1. Loading one OPTEE at address 0x06280000 successfully by following the guidance attached below.
2. In the file <path-to-workspace>/optee_os/core/arch/arm/plat-vexpress/conf.mk
              change "CFG_TZDRAM_START ?=0x06281000" to "CFG_TZDRAM_START ?= 0x08002000"
3. In the file <path-to-workspace>//trusted-firmware-a/fdts/optee_sp_manifest.dts
             change  "load-address = <0x6280000>" to "load-address = <0x8001000>"
4. Compile OPTEE
      CROSS_COMPILE64=aarch64-none-elf- PLATFORM=vexpress PLATFORM_FLAVOR=fvp CFG_ARM_GICV3=y CFG_CORE_SEL2_SPMC=y CFG_ARM64_core=y CFG_USER_TA_TARGETS=ta_arm64 CFG_TEE_CORE_DEBUG=y CFG_TEE_CORE_LOG_LEVEL=4 CFG_TEE_BENCHMARK=n CFG_WITH_STATS=y make
5. Compile ATF
      make CROSS_COMPILE=aarch64-none-elf- SPD=spmd CTX_INCLUDE_EL2_REGS=1 PLAT=fvp BL33=../tf-a-tests/build/fvp/debug/tftf.bin DEBUG=1 BL32=../hafnium/out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ARM_ARCH_MINOR=4 SP_LAYOUT_FILE=sp_layout.json ARM_SPMC_MANIFEST_DTS=plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts all fip
6. Run FVP
     ../fvp/Base_RevC_AEMv8A_pkg/models/Linux64_GCC-6.4/FVP_Base_RevC-2xAEMv8A -C pctl.startup=0.0.0.0 -C cluster0.NUM_CORES=4 -C cluster1.NUM_CORES=4 -C bp.secure_memory=1 -C bp.secureflashloader.fname=trusted-firmware-a/build/fvp/debug/bl1.bin -C bp.flashloader0.fname=trusted-firmware-a/build/fvp/debug/fip.bin -C cluster0.has_arm_v8-4=1 -C cluster1.has_arm_v8-4=1 -C cache_state_modelled=0 -C bp.pl011_uart0.out_file=fvp-uart0.log -C bp.pl011_uart1.out_file=fvp-uart1.log -C bp.vis.disable_visualisation=1
7. The error message from fvp-uart0.log (fvp-uart1.log is empty)
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL1: Built : 09:39:24, Feb 25 2021
INFO:    BL1: RAM 0x4035000 - 0x403c000
INFO:    Loading image id=31 at address 0x4001010
INFO:    Image id=31 loaded: 0x4001010 - 0x4001234
INFO:    FCONF: Config file with image ID:31 loaded at address = 0x4001010
INFO:    Loading image id=24 at address 0x4001300
INFO:    Image id=24 loaded: 0x4001300 - 0x4001466
INFO:    FCONF: Config file with image ID:24 loaded at address = 0x4001300
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0x4022000
INFO:    Image id=1 loaded: 0x4022000 - 0x402bb49
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0x4022000
INFO:    SPSR = 0x3c5
INFO:    FCONF: Reading FW_CONFIG firmware configuration file from: 0x4001010
INFO:    FCONF: Reading firmware configuration information for: dyn_cfg
INFO:    FCONF: Reading TB_FW firmware configuration file from: 0x4001300
INFO:    FCONF: Reading firmware configuration information for: arm_sp
NOTICE:  BL2: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL2: Built : 09:39:26, Feb 25 2021
INFO:    BL2: Doing platform setup
INFO:    Configuring TrustZone Controller
INFO:    Total 4 regions set.
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0x4003000
INFO:    Image id=3 loaded: 0x4003000 - 0x401722c
INFO:    BL2: Loading image id 23
INFO:    Loading image id=23 at address 0x82000000
INFO:    Image id=23 loaded: 0x82000000 - 0x82002348
INFO:    BL2: Loading image id 25
INFO:    Loading image id=25 at address 0x4001300
INFO:    Image id=25 loaded: 0x4001300 - 0x4001348
INFO:    BL2: Loading image id 4
INFO:    Loading image id=4 at address 0x6000000
INFO:    Image id=4 loaded: 0x6000000 - 0x601e250
INFO:    BL2: Skip loading image id 21
INFO:    BL2: Skip loading image id 22
INFO:    BL2: Loading image id 26
INFO:    Loading image id=26 at address 0x4001500
INFO:    Image id=26 loaded: 0x4001500 - 0x4001acf
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x88000000
INFO:    Image id=5 loaded: 0x88000000 - 0x8800b2b0
INFO:    BL2: Loading image id 34
INFO:    Loading image id=34 at address 0x6280000
INFO:    Image id=34 loaded: 0x6280000 - 0x62e2050
NOTICE:  BL1: Booting BL31
INFO:    Entry point address = 0x4003000
INFO:    SPSR = 0x3cd
INFO:    FCONF: Reading HW_CONFIG firmware configuration file from: 0x82000000
INFO:    FCONF: Reading firmware configuration information for: cpu_timer
INFO:    FCONF: Reading firmware configuration information for: uart_config
INFO:    FCONF: Reading firmware configuration information for: topology
INFO:    FCONF: Reading firmware configuration information for: gicv3_config
NOTICE:  BL31: v2.4(debug):v2.4-79-gb4b23c780-dirty
NOTICE:  BL31: Built : 09:39:31, Feb 25 2021
INFO:    GICv3 with legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    BL31: Initializing runtime services
INFO:    SPM Core setup done.
INFO:    BL31: Initializing BL32
NOTICE: Initialising hafnium
INFO: text: 0x6000000 - 0x601a000
INFO: rodata: 0x601a000 - 0x601e000
INFO: data: 0x601e000 - 0x60a0000
INFO: Supported bits in physical address: 40
INFO: Stage 2 has 3 page table levels with 2 pages at the root.
INFO:    SPM: 0x84000069 0x0 0x0 0x0 0x0 0x0 0x0 0x0
INFO: Memory range:  0x6000000 - 0x7ffffff
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x0 0x6001340 0x601e000 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10300 0x6001340 0x601e0a8 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10200 0x6001340 0x601e090 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10100 0x6001340 0x601e078 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x10000 0x6001340 0x601e060 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x300 0x6001340 0x601e048 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x200 0x6001340 0x601e030 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO:    SPM: 0x8400006f 0xffffffff8000ffff 0x0 0x1 0x100 0x6001340 0x601e018 0x0
INFO:    In case FFA_MSG_SEND_DIRECT_REQ_SMC32, branch1 (this message is added by myself)
INFO: Loaded primary VM with 8 vCPUs, entry at 0x8001000.
INFO: Hafnium initialisation completed
WARNING: Stage-2 page fault: pc=0x8002000, vmid=32769, vcpu=0, vaddr=0x8002000, ipaddr=0x0, mode=0x4
NOTICE: Injecting Instruction Abort exception into VM32769.
WARNING: Stage-2 page fault: pc=0x200, vmid=32769, vcpu=0, vaddr=0x200, ipaddr=0x200, mode=0x4
NOTICE: Injecting Instruction Abort exception into VM32769.
WARNING: Stage-2 page fault: pc=0x200, vmid=32769, vcpu=0, vaddr=0x200, ipaddr=0x200, mode=0x4
NOTICE: Injecting Instruction Abort exception into VM32769.
WARNING: Stage-2 page fault: pc=0x200, vmid=32769, vcpu=0, vaddr=0x200, ipaddr=0x200, mode=0x4
NOTICE: Injecting Instruction Abort exception into VM32769.
.
.
.
(infinite loop . . .)
============The guidance from Oliver Deprez to load one OPTEE============
# NOTE: make sure gcc-arm-9.2-2019.12-x86_64-aarch64-none-elf/bin is in PATH
mkdir workspace; cd workspace
# Hafnium
# https://review.trustedfirmware.org/plugins/gitiles/hafnium/hafnium/+/HEAD/do...
git clone https://git.trustedfirmware.org/hafnium/hafnium.git; cd hafnium git fetch "https://review.trustedfirmware.org/hafnium/hafnium" refs/changes/11/6011/16 && git checkout FETCH_HEAD git checkout -b spmc git submodule update --init make PROJECT=reference cd ..
# OPTEE
# https://optee.readthedocs.io/en/latest/building/prerequisites.html#prerequis...
git clone https://git.trustedfirmware.org/OP-TEE/optee_os.git -b ffa_rel_proto; cd optee_os
# Adjust TEE start address
perl -pi -e 's/CFG_TZDRAM_START ?= 0x06280000/CFG_TZDRAM_START ?= 0x06281000/' core/arch/arm/plat-vexpress/conf.mk
CROSS_COMPILE64=aarch64-none-elf- PLATFORM=vexpress PLATFORM_FLAVOR=fvp CFG_ARM_GICV3=y CFG_CORE_SEL2_SPMC=y CFG_ARM64_core=y CFG_USER_TA_TARGETS=ta_arm64 CFG_TEE_CORE_DEBUG=y CFG_TEE_CORE_LOG_LEVEL=4 CFG_TEE_BENCHMARK=n CFG_WITH_STATS=y make cd ..
# TF-A-tests
git clone https://git.trustedfirmware.org/TF-A/tf-a-tests.git; cd tf-a-tests make CROSS_COMPILE=aarch64-none-elf- PLAT=fvp DEBUG=1 TESTS=spm -j4 cd ..
# TF-A
#
git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git; cd trusted-firmware-a git checkout b4b23c780c0763ce01ac21de1a0e79287677405f -b spmc
#{
#    "op-tee" : {
#       "image": "../optee_os/out/arm-plat-vexpress/core/tee-pager_v2.bin",
# "pm": "fdts/optee_sp_manifest.dts"
#    }
#}
echo \
"{ \
    "op-tee" : { \
    "image": "../optee_os/out/arm-plat-vexpress/core/tee-pager_v2.bin", \
    "pm": "fdts/optee_sp_manifest.dts" \ } }" > sp_layout.json
make CROSS_COMPILE=aarch64-none-elf- SPD=spmd CTX_INCLUDE_EL2_REGS=1 PLAT=fvp BL33=../tf-a-tests/build/fvp/debug/tftf.bin DEBUG=1 BL32=../hafnium/out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ARM_ARCH_MINOR=4 SP_LAYOUT_FILE=sp_layout.json ARM_SPMC_MANIFEST_DTS=plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts all fip cd ..
# Run FVP
../fvp/Base_RevC_AEMv8A_pkg/models/Linux64_GCC-6.4/FVP_Base_RevC-2xAEMv8A -C pctl.startup=0.0.0.0 -C cluster0.NUM_CORES=4 -C cluster1.NUM_CORES=4 -C bp.secure_memory=1 -C bp.secureflashloader.fname=trusted-firmware-a/build/fvp/debug/bl1.bin -C bp.flashloader0.fname=trusted-firmware-a/build/fvp/debug/fip.bin -C cluster0.has_arm_v8-4=1 -C cluster1.has_arm_v8-4=1 -C cache_state_modelled=0 -C bp.pl011_uart0.out_file=fvp-uart0.log -C bp.pl011_uart1.out_file=fvp-uart1.log -C bp.vis.disable_visualisation=1
=============================
I appreciate for any possible help!
Thanks
Yinhua
--
Hafnium mailing list
Hafnium@lists.trustedfirmware.orgmailto:Hafnium@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
--
Hafnium mailing list
Hafnium@lists.trustedfirmware.orgmailto:Hafnium@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
--
Hafnium mailing list
Hafnium@lists.trustedfirmware.orgmailto:Hafnium@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium