Hi,
Thanks for your efforts for this query. I have tried your solution. Although the Hafnium code could be built successfully, it does not work when the boot flow run into the entry of the Hafnium.
I tried to debug the error with trace32 tool. And I confirmed that the Hafnium code has been loaded over 34G of free address space. However, when PC register point to the start address of Hafnium, the code data is displayed as udf #0x0.
It may be a page table problem, and we are still locating the problem reason.
Regards,
Mei, Jianqiang.
------------------------------------------------------------------
发件人：Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
发送时间：2022年12月21日(星期三) 18:02
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>; 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 黄明(连一) <hm281385(a)alibaba-inc.com>
主 题：[Hafnium] Re: Hafnium load-address
Hi,
Sorry for the delay in replying to this query.
After few trials I ended with this possible solution:
https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 <https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 >
This should permit loading the Hafnium image to an address up to 48 bits (and by updating project/reference/BUILD.gn origin_address per your suggestion).
It is possible to further extend beyond 48 bits (e.g. 52 bits which is the maximum permitted VA using FEAT_LPA2).
Let me know if this works for you, then I can proceed further with polishing this change.
Regards,
Olivier.
________________________________
From: 梅建强(禹夜) via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 11 December 2022 09:58
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>; 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 黄明(连一) <hm281385(a)alibaba-inc.com>
Subject: [Hafnium] Hafnium load-address
Hello, expert
Firstly, thanks for your answer some time ago. That's useful to our project.We combined the Hafnium (run at S-EL2) and the optee (run at S-EL1), then modified the configuration to adapt to our own environment. Now we can start the os normally.
Now we want to optimize a technical point, which is to load and run hafnium over 34G of free address space. This requires hafnium to support 64-bit load-address.
I tried to modify the configuration in BUILD.gn as follows:
origin_address = "0x880000000"
And I encountered the build error:
FAILED: secure_aem_v8a_fvp_clang/hafnium.elf ld.lld -pie --gc-sections --defsym=ORIGIN_ADDRESS=0x880000000 -O2 --icf=all --fatal-warnings --color-diagnostics -T /home/yuye/hafnium/Hafnium/build/image/image.ld -o secure_aem_v8a_fvp_clang/hafnium.elf --start-group @secure_aem_v8a_fvp_clang/hafnium.elf.rsp --end-group ld.lld: error: secure_aem_v8a_fvp_clang/obj/src/arch/aarch64/entry.entry.o:(.init.entry+0xC8): relocation R_AARCH64_ABS32 out of range: 36507222016 is not in [-2147483648, 4294967295]; references ORIGIN_ADDRESS >>> defined in -defsym:1 ninja: build stopped: subcommand failed. Makefile:70: recipe for target 'all' failed make: *** [all] Error 1 + cp out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ../../V2/Ali_M1_Public/Nonosi/Platform/Ali/M1/Bin/ cp: cannot stat 'out/reference/secure_aem_v8a_fvp_clang/hafnium.bin': No such file or directory
In this case, it seems that the new address causes some relative addressing to reach too far. Is there any solutions or supports for this issue currently in the community?
Regards,
Mei, Jianqiang.
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi,
At present, I suspect that this may be a hafnium problem,
secure_storage CA/TA may use over 34G address space before communication.
May I ask, does hafnium currently support the configuration of the page table of 0x8 80000000-xxxxxxxxx address range?
If yes, how to configure it?
The error log is as follows:
Current share states: SHARE 0x0 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7 SHARE 0x1 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7 SHARE 0x2 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7 WARNING: Stage-2 page fault: pc=0x40032106, vmid=0x8001, vcpu=13, vaddr=0x40046ce0, ipaddr=0x8a9b28ce0, mode=0x81 0x63 NOTICE: Injecting Data Abort exception into VM 0x8001. D/TC:013 0 abort_handler:550 [abort] abort in User mode (TA will panic) E/TC:??? 0 E/TC:??? 0 User mode data-abort at address 0x40046ce0 (translation fault) E/TC:??? 0 esr 0x94020007 ttbr0 0x20000f03061a0 ttbr1 0x00000000 cidr 0x0 E/TC:??? 0 cpu #13 cpsr 0x00000130 E/TC:??? 0 x0 000000004003e348 x1 000000004003e349 E/TC:??? 0 x2 0000000040046ce0 x3 000000004003e348 E/TC:??? 0 x4 0000000040036088 x5 0000000000000000 E/TC:??? 0 x6 0000000000000000 x7 0000000040042e28 E/TC:??? 0 x8 0000000000000000 x9 0000000000000000 E/TC:??? 0 x10 0000000000000000 x11 0000000000000000 E/TC:??? 0 x12 0000000000000000 x13 0000000040042e28 E/TC:??? 0 x14 00000000400213cf x15 0000000000000000 E/TC:??? 0 x16 0000000000000000 x17 0000000000000000 E/TC:??? 0 x18 0000000000000000 x19 0000000000000000 E/TC:??? 0 x20 0000000000000000 x21 0000000000000000 E/TC:??? 0 x22 0000000000000000 x23 0000000000000000 E/TC:??? 0 x24 0000000000000000 x25 0000000000000000 E/TC:??? 0 x26 0000000000000000 x27 0000000000000000 E/TC:??? 0 x28 0000000000000000 x29 0000000000000000 E/TC:??? 0 x30 0000000000000000 elr 0000000040032106 E/TC:??? 0 sp_el0 0000000040042f80 E/LD: Status of TA f4e750bb-1437-4fbf-8785-8d3580c34994 E/LD: arch: arm E/LD: region 0: va 0x40006000 pa 0xf0404000 size 0x002000 flags rw-s (ldelf) E/LD: region 1: va 0x40008000 pa 0xf0406000 size 0x011000 flags r-xs (ldelf) E/LD: region 2: va 0x40019000 pa 0xf0417000 size 0x001000 flags rw-s (ldelf) E/LD: region 3: va 0x4001a000 pa 0xf0418000 size 0x004000 flags rw-s (ldelf) E/LD: region 4: va 0x4001e000 pa 0xf041c000 size 0x001000 flags r--s E/LD: region 5: va 0x4001f000 pa 0x00001000 size 0x017000 flags r-xs [0] E/LD: region 6: va 0x40036000 pa 0x00018000 size 0x00c000 flags rw-s [0] E/LD: region 7: va 0x40042000 pa 0xf0440000 size 0x001000 flags rw-s (stack) E/LD: region 8: va 0x40043000 pa 0x8b9101620 size 0x003000 flags rw-- (param) E/LD: region 9: va 0x40046000 pa 0x8a9b28ce0 size 0x001000 flags rw-- (param) E/LD: [0] f4e750bb-1437-4fbf-8785-8d3580c34994 @ 0x4001f000 E/LD: Call stack
Regards,
Yuye
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年2月4日(星期六) 11:40
收件人：op-tee <op-tee(a)lists.trustedfirmware.org>
抄 送：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 高海源(码源) <haiyuan.ghy(a)alibaba-inc.com>; 王一蒙(北见) <wym389994(a)alibaba-inc.com>
主 题：OPTEE TA Crash
Hi,
Does anyone have a good solution to this problem？
https://github.com/OP-TEE/optee_os/issues/5803 <https://github.com/OP-TEE/optee_os/issues/5803 >
regards,
yuye
Hi, expert
I want to confirm a problem about using hafnium to load spmc payload.
That is, Dose hafnium supports hyp_loaded for spmc payload or ffa_partion (such as optee_os) in the boot phase?
If the community confirms that it does not support it, I wonder, what might be the problem with doing so?
I see the following code in hanfnium：
if (manifest->vm[i].is_ffa_partition &&
!manifest->vm[i].is_hyp_loaded) {
TRY(parse_ffa_partition_package(stage1_locked, &vm_node,
&manifest->vm[i], vm_id,
ppool));
} else {
TRY(parse_vm(&vm_node, &manifest->vm[i], vm_id));
}
It seems that hafnium has no handling for both is_ffa_partition and is_hyp_loaded.
regards,
yuye
Hi,
Fyi we merged this gn tool prebuilt version update
https://review.trustedfirmware.org/q/topic:%22od%252Fgn-update-1901%22+(sta…
Essentially more recent sources were used to build the x86 host version (as the tool hasn't been updated since 2018), and this aligns to the version used on aarch64 host.
'make format' uses gn to format .gn build files, and as a consequence new formatting rules apply with the new tool version.
The side effect is that changes already submitted in the hafnium dashboards would have to be rebased, and run make format again to pass the CI.
Regards,
Olivier.
Hi,
Currently each patch submission automatically starts two jenkins jobs for cloning, building, running tests and static checks.
The longest of the two jobs can run for 30mn - 1h depending on server load.
A 10 patches stack queues as many of such jobs and it can be a long process to get all votes with a significant load applied to the servers.
Along with this the Allow-CI+1 label can be used by maintainers to re-trigger a job either because it failed, or the results/logs were flushed.
Per discussion with various stakeholders we come to a conclusion it would be preferable to only use the Allow-CI label and discard the automatic trigger for each and every patch.
Similarly to TF-A, a change submitted by a developer requires a maintainer to apply the Allow-CI+1 label to build and run tests.
For a large patch stack, the expectation is at least the top patch must pass the CI run before merging, but not necessarily all intermediate patches.
It's the maintainer discretion to apply the label at different places in the patch stack to get intermediate results as required.
I intend to submit a change shortly to adopt this new policy.
Let me know if any concern.
Regards,
Olivier.
Hello, expert
Firstly, thanks for your answer some time ago. That's useful to our project.We combined the Hafnium (run at S-EL2) and the optee (run at S-EL1), then modified the configuration to adapt to our own environment. Now we can start the os normally.
Now we want to optimize a technical point, which is to load and run hafnium over 34G of free address space. This requires hafnium to support 64-bit load-address.
I tried to modify the configuration in BUILD.gn as follows:
origin_address = "0x880000000"
And I encountered the build error:
FAILED: secure_aem_v8a_fvp_clang/hafnium.elf ld.lld -pie --gc-sections --defsym=ORIGIN_ADDRESS=0x880000000 -O2 --icf=all --fatal-warnings --color-diagnostics -T /home/yuye/hafnium/Hafnium/build/image/image.ld -o secure_aem_v8a_fvp_clang/hafnium.elf --start-group @secure_aem_v8a_fvp_clang/hafnium.elf.rsp --end-group ld.lld: error: secure_aem_v8a_fvp_clang/obj/src/arch/aarch64/entry.entry.o:(.init.entry+0xC8): relocation R_AARCH64_ABS32 out of range: 36507222016 is not in [-2147483648, 4294967295]; references ORIGIN_ADDRESS >>> defined in -defsym:1 ninja: build stopped: subcommand failed. Makefile:70: recipe for target 'all' failed make: *** [all] Error 1 + cp out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ../../V2/Ali_M1_Public/Nonosi/Platform/Ali/M1/Bin/ cp: cannot stat 'out/reference/secure_aem_v8a_fvp_clang/hafnium.bin': No such file or directory
In this case, it seems that the new address causes some relative addressing to reach too far. Is there any solutions or supports for this issue currently in the community?
Regards,
Mei, Jianqiang.
Hi,
> Currently we bypass smmu and has finished boot process, later we will discuss whether there is a scenario for using smmu in the project.
That's great news!
> Yes, we use it on the server and need to support LPA.
Just as a matter of clarifying, Hafnium treats FEAT_LPA by restricting the physical address space to 48 bits:
https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/m…
The reason is that with FEAT_LPA, only the 16KB/64KB translation granules are supported, and Hafnium only supports the 4KB translation granule.
It means the normal world shall not attempt providing physical addresses (e.g. in memory sharing operations to the SPMC) where the PA uses an address size greater than 48 bits.
Is this a reasonable assumption when integrating in your system?
Secondly we were reported a slight issue when applying this restriction by Jens @ Linaro.
I appreciate the change below might be submitted shortly for upstream.
https://github.com/jenswi-linaro/hafnium/commit/659c79d5eacf32e8f5fcb1a6403…
Regards,
Olivier.
From: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Sent: 06 December 2022 10:49
To: Olivier Deprez <Olivier.Deprez(a)arm.com>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Cc: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
Subject: 回复：回复：SMMU Configuration
Hi Olivier,
Thank you very much for your reply and suggestion.
Currently we bypass smmu and has finished boot process, later we will discuss whether there is a scenario for using smmu in the project.
Other question are you relying on FEAT_LPA/FEAT_LPA2?
---Yes, we use it on the server and need to support LPA.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年12月1日(星期四) 17:12
收件人：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
主 题：Re: 回复：SMMU Configuration
Hi,
> Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24，
Thanks for confirming.
> Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage?
No, as observed here:
https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/a…
But this is an interesting input to us, as we've validated the driver with a low STEs count and this only required a linear table.
So this is an improvement we may think about.
> hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a
> yes，i have set memory range security attributes by the TZC controller.
Alright.
Before going further, do you confirm you have use cases requiring this SMMU driver/component?
In particular do you have devices upstream to this SMMU generating transactions through a secure stream ID?
e.g. this can be an SPI peripheral only accessible from the secure world?
What I mean here is there is no point in including the driver (and consuming a lot of resources) if there is no use case to fulfill eventually?
If there are, then yes, we can go ahead and find solutions to enable such use cases.
For the sake of experiment, did you attempt increasing the number of heap pages beyond 262144?
I don't have a clear idea of the memory requirements for your platform. In the reference code, the default is 180 heap pages, but I assume this can increase a lot depending on the memory size, secure partitions S2 mappings etc.
Other question are you relying on FEAT_LPA/FEAT_LPA2?
Regards,
Olivier.
From: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Sent: 01 December 2022 04:00
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
Subject: 回复：SMMU Configuration
Hi，
1、Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24， Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage？
2、 hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a5c
3、 yes，i have set memory range security attributes by the TZC controller.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年11月30日(星期三) 17:17
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; Olivier Deprez <Olivier.Deprez(a)arm.com>
抄 送：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: SMMU Configuration
Hi,
Another out of curiosity question about:
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
Hafnium as SPMC in the secure side must allocate data structures only from secure memory.
Do you confirm this memory range is marked secure by the TZASC controller?
Regards,
Olivier.
From: Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 30 November 2022 10:13
To: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
Subject: [Hafnium] Re: SMMU Configuration
Hi,
1/ About:
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24?
In which case the stream table size would grow up to 1GB.
Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation.
Can you confirm you have this change in your tree:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a06…
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 28 November 2022 10:48
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Subject: SMMU Configuration
Hi,
when I enabled SMMU by set the config file as follows：
aarch64_toolchains("secure_aem_v8a_fvp") {
cpu = "cortex-a57"
origin_address = "0x06000000"
boot_flow = "//src/boot_flow:spmc"
console = "//src/arch/aarch64/pl011"
iommu = "//src/arch/aarch64/arm_smmuv3"
gic_version = 3
gicd_base_address = "0x2e000000"
gicr_base_address = "0x2e140000"
gicr_frames = 8
heap_pages = 100000
max_cpus = 8
max_vms = 16
# branch_protection = "standard"
toolchain_args = {
plat_ffa = "//src/arch/aarch64/plat/ffa:spmc"
plat_psci = "//src/arch/aarch64/plat/psci:spmc"
plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3"
plat_prng = "//src/arch/aarch64/plat/prng:prng"
secure_world = "1"
pl011_base_address = "0x2A400000"
smmu_base_address = "0x3FC00000"
smmu_memory_size = "0x400000"
# enable_mte = "1"
plat_log_level = "LOG_LEVEL_VERBOSE"
}
}
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA
VERBOSE: SMMUv3: write to (S_)CR0
VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits
VERBOSE: SMMUv3: Total CMDQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
VERBOSE: SMMUv3: write to (S_)CMDQ_BASE
VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD
VERBOSE: SMMUv3: Total EVTQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ
VERBOSE: SMMUv3: write to (S_)EVTQ_BASE
VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
ERROR: SMMUv3: Could not allocate memory for stream table entries
ERROR: SMMUv3: Failed to initialize driver
Panic: Could not initialize IOMMUs.
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi,
> Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24，
Thanks for confirming.
> Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage?
No, as observed here:
https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/a…
But this is an interesting input to us, as we've validated the driver with a low STEs count and this only required a linear table.
So this is an improvement we may think about.
> hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a
> yes，i have set memory range security attributes by the TZC controller.
Alright.
Before going further, do you confirm you have use cases requiring this SMMU driver/component?
In particular do you have devices upstream to this SMMU generating transactions through a secure stream ID?
e.g. this can be an SPI peripheral only accessible from the secure world?
What I mean here is there is no point in including the driver (and consuming a lot of resources) if there is no use case to fulfill eventually?
If there are, then yes, we can go ahead and find solutions to enable such use cases.
For the sake of experiment, did you attempt increasing the number of heap pages beyond 262144?
I don't have a clear idea of the memory requirements for your platform. In the reference code, the default is 180 heap pages, but I assume this can increase a lot depending on the memory size, secure partitions S2 mappings etc.
Other question are you relying on FEAT_LPA/FEAT_LPA2?
Regards,
Olivier.
From: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Sent: 01 December 2022 04:00
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
Subject: 回复：SMMU Configuration
Hi，
1、Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24， Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage？
2、 hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a5c
3、 yes，i have set memory range security attributes by the TZC controller.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年11月30日(星期三) 17:17
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; Olivier Deprez <Olivier.Deprez(a)arm.com>
抄 送：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: SMMU Configuration
Hi,
Another out of curiosity question about:
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
Hafnium as SPMC in the secure side must allocate data structures only from secure memory.
Do you confirm this memory range is marked secure by the TZASC controller?
Regards,
Olivier.
From: Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 30 November 2022 10:13
To: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
Subject: [Hafnium] Re: SMMU Configuration
Hi,
1/ About:
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24?
In which case the stream table size would grow up to 1GB.
Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation.
Can you confirm you have this change in your tree:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a06…
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 28 November 2022 10:48
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Subject: SMMU Configuration
Hi,
when I enabled SMMU by set the config file as follows：
aarch64_toolchains("secure_aem_v8a_fvp") {
cpu = "cortex-a57"
origin_address = "0x06000000"
boot_flow = "//src/boot_flow:spmc"
console = "//src/arch/aarch64/pl011"
iommu = "//src/arch/aarch64/arm_smmuv3"
gic_version = 3
gicd_base_address = "0x2e000000"
gicr_base_address = "0x2e140000"
gicr_frames = 8
heap_pages = 100000
max_cpus = 8
max_vms = 16
# branch_protection = "standard"
toolchain_args = {
plat_ffa = "//src/arch/aarch64/plat/ffa:spmc"
plat_psci = "//src/arch/aarch64/plat/psci:spmc"
plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3"
plat_prng = "//src/arch/aarch64/plat/prng:prng"
secure_world = "1"
pl011_base_address = "0x2A400000"
smmu_base_address = "0x3FC00000"
smmu_memory_size = "0x400000"
# enable_mte = "1"
plat_log_level = "LOG_LEVEL_VERBOSE"
}
}
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA
VERBOSE: SMMUv3: write to (S_)CR0
VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits
VERBOSE: SMMUv3: Total CMDQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
VERBOSE: SMMUv3: write to (S_)CMDQ_BASE
VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD
VERBOSE: SMMUv3: Total EVTQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ
VERBOSE: SMMUv3: write to (S_)EVTQ_BASE
VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
ERROR: SMMUv3: Could not allocate memory for stream table entries
ERROR: SMMUv3: Failed to initialize driver
Panic: Could not initialize IOMMUs.
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi,
1/ About:
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24?
In which case the stream table size would grow up to 1GB.
Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation.
Can you confirm you have this change in your tree:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a06…
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 28 November 2022 10:48
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Subject: SMMU Configuration
Hi,
when I enabled SMMU by set the config file as follows：
aarch64_toolchains("secure_aem_v8a_fvp") {
cpu = "cortex-a57"
origin_address = "0x06000000"
boot_flow = "//src/boot_flow:spmc"
console = "//src/arch/aarch64/pl011"
iommu = "//src/arch/aarch64/arm_smmuv3"
gic_version = 3
gicd_base_address = "0x2e000000"
gicr_base_address = "0x2e140000"
gicr_frames = 8
heap_pages = 100000
max_cpus = 8
max_vms = 16
# branch_protection = "standard"
toolchain_args = {
plat_ffa = "//src/arch/aarch64/plat/ffa:spmc"
plat_psci = "//src/arch/aarch64/plat/psci:spmc"
plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3"
plat_prng = "//src/arch/aarch64/plat/prng:prng"
secure_world = "1"
pl011_base_address = "0x2A400000"
smmu_base_address = "0x3FC00000"
smmu_memory_size = "0x400000"
# enable_mte = "1"
plat_log_level = "LOG_LEVEL_VERBOSE"
}
}
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA
VERBOSE: SMMUv3: write to (S_)CR0
VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits
VERBOSE: SMMUv3: Total CMDQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
VERBOSE: SMMUv3: write to (S_)CMDQ_BASE
VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD
VERBOSE: SMMUv3: Total EVTQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ
VERBOSE: SMMUv3: write to (S_)EVTQ_BASE
VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
ERROR: SMMUv3: Could not allocate memory for stream table entries
ERROR: SMMUv3: Failed to initialize driver
Panic: Could not initialize IOMMUs.
Cc Hafnium list.
________________________________
From: Olivier Deprez <Olivier.Deprez(a)arm.com>
Sent: 25 November 2022 17:41
To: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Subject: Re: 回复：run hafnium as spmc at s-el2
Hi,
See answers inline [OD]
Regards,
Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 25 November 2022 03:18
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Subject: 回复：run hafnium as spmc at s-el2
Hello，expert
I do intend to use OP-TEE as SEL1 payload / secure partition with hafnium running on SEL2 and run linux in the normal world.
In my environment，when TF-A and Hafnium boot, only a single primary core runs. After the os starts, the secondary core is booted by bl31.
Now, I have some other questions ：
1）When entering hafnium to initialize gic, I get the following error (the first two lines are my debug prints) :
populate_redist_base_addrs:375,typer_reg:844425014019047,current_rdist_frame:2e200000
populate_redist_base_addrs:375,typer_reg:0,current_rdist_frame:2e220000
ERROR: Data abort: pc=0xff20558c, esr=0x96000007, ec=0x25, far=0x2e240008
The gic appears to be missing the corresponding page table configuration.
Hafnium runs on s-el2, and according to the arm specification, it has a set of page tables for itself and only does stage 1 translation.
So how to configure the page table for hafnium to include the address space where the gic resides?
[OD] Are you using the Base FVP platform?
The SPMC run address suggests this might be an Infra/Neoverse platform?
Did you take care of creating a new platform, or update the FVP defaults for the GICD/GICR system addresses?
https://git.trustedfirmware.org/hafnium/project/reference.git/tree/BUILD.gn…
2）Is smmu necessary for hanfium to run as SPMC, how much running memory does hafnium need？
And how to specify the address space where its running memory resides?
[OD] This depends first if your platform implements an SMMU and it supports secure S2 translation (from >=SMMUv3.2).
Second condition is if you need to protect VMs from a peripheral upstream from this SMMU.
If not, you can omit the SMMU driver by removing the iommu property:
https://git.trustedfirmware.org/hafnium/project/reference.git/tree/BUILD.gn…
3）When TF-A and Hafnium boot, only a single primary core runs. How to configure the cpu core to boot the hafnium?
[OD] When the platform resets and starts, a single core is up (aka the primary core) and TF-A boots on this core through BL1/BL2/BL31 stages.
From there BL31 launches BL32/Hafnium on the same core.
There is no configurability from a SW perspective. It is up to the platform itself to tie onto which core the system boots.
If running on a model, that *may* be controlled by a model parameter.
In addition, there are multiple cores at runtime. How does hanfium switch cores to switch the corresponding secure partions?
How does core switching relate to vCPU switching？
[OD] Hafnium is controlled by a normal world driver e.g. the linux upstream FF-A driver: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drive… and a TEE driver e.g. OP-TEE driver https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drive…
The primary scheduler is linux' and Hafnium is not performing scheduling decisions.
As an example, a direct request on CPU1 emitted from the normal world through an SMC, is processed by the SPMC and delivered to the SP(OP-TEE) on vCPU1.
Thank you very much～
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年11月25日(星期五) 03:24
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: run hafnium as spmc at s-el2
Hi Mei Jianqiang,
> In my environment, hafnium is running as spmc in s-el2, and my question is as follows:
Can you tell a bit more on your setup, if possible, do you intend to use OP-TEE as SEL1 payload / secure partition?
Are you running linux in the normal world?
> 1) In the spmc_optee_sp_manifest.dts file, there is an attribute load_address = <0x0 0xFF200000>.
> What do the two parts of load_address mean respectively? Any details on other configurations in the dts file?
In the TF-A tree plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts is a sample device tree consumed by Hafnium/SPMC to describe the system properties and declare secure partitions.
Please refer to https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit…
The load_address field specifies the location at which the SPMC finds a 'partition package'. It consists of a partition's DT blob and a partition image.
https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit…
In this sample 0x0 is the high 32b part of the physical address and 0xFF200000 the lower 32 bits.
> 2) If only one core is available during the startup of hafnium, how to configure it?
> In this case, when I set CPUS_PER_CLUSTER 1 in fvp-defs.dtsi file of ATF reference code, the building error is: Duplicate node name /cpus/cpu@20000. Why?
I don't believe you should do change directly in this file, but rather provide the system topology through the TF-A command line by using FVP_MAX_CPUS_PER_CLUSTER/FVP_CLUSTER_COUNT/FVP_MAX_PE_PER_CPU
The device tree should describe cpus as they exist when the system is booted at run-time.
When TF-A and Hafnium boot, only a single primary core runs anyways so is this really what you intend to do?
Regards,
Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 24 November 2022 11:33
To: hafnium-owner <hafnium-owner(a)lists.trustedfirmware.org>
Subject: run hafnium as spmc at s-el2
Hello expert,
I'm a developer at Alibaba Cloud, and I'm having some problems using hafnium.
In my environment, hafnium is running as spmc in s-el2, and my question is as follows:
1) In the spmc_optee_sp_manifest.dts file, there is an attribute load_address = <0x0 0xFF200000>.
What do the two parts of load_address mean respectively? Any details on other configurations in the dts file?
2) If only one core is available during the startup of hafnium, how to configure it?
In this case, when I set CPUS_PER_CLUSTER 1 in fvp-defs.dtsi file of ATF reference code, the building error is: Duplicate node name /cpus/cpu@20000. Why?
Looking forward to your reply. Thank you very much~
