Hi,
I'm testing with Hafnium as SPMC at S-EL2 and OP-TEE as an SP at S-EL1 on
QEMU v7.0.0. I've run into a few problems and fixed most of them.
I believe the setup is similar to what Shiju is using in this mail thread
https://lists.trustedfirmware.org/archives/list/hafnium@lists.trustedfirmwa…
My setup can be duplicated with:
repo init -u https://github.com/jenswi-linaro/manifest.git -m qemu_v8.xml \
-b qemu_sel2
repo sync -j8
(cd hafnium && git submodule init && git submodule update)
cd build
make -j8 toolchains
make -j8 all
make run-only
With this xtest -x 1034 passes, xtest 1034 often causes
ERROR: Data abort: pc=0xe1198b8, esr=0x96000006, ec=0x25, far=0x9c
Panic: EL2 exception
Xtest runs dreadfully slow, I haven't investigated why yet, but at
least it works.
This is based on patches provided by Olivier at:
[1] https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/16412/2
[2] https://review.trustedfirmware.org/c/hafnium/hafnium/+/16323/7
I've also encountered the problem cache maintenance problem Shiju
described in the mail thread above:
NOTICE: Trapped access to system register write: op0=1, op1=0, crn=7,
crm=14, op2=2, rt=11.
It can be worked around by compiling OP-TEE with
CFG_CORE_WORKAROUND_NSITR_CACHE_PRIME=n. I'm pretty sure we do dcache
clean+inv elsewhere so I'm surprised it fails here. Is Hafnium expected
to block dcache clean+inv?
For Hafnium I've added two patches on top of [2], available at
https://github.com/jenswi-linaro/hafnium/tree/qemu_sel2:
- 79b4d2cbe06e SPMC: add missing ME initialization for secondary cores
- 659c79d5eacf feat(mm): fix FEAT_LPA workaround
For TF-A I've added a few patches on top of [1], available at
https://github.com/jenswi-linaro/arm-trusted-firmware/tree/qemu_sel2:
- a040396cae9e feat(qemu): add tos-fw-config for sel2 spmc
- 4f7d91723485 fix(qemu): change TOS_FW_CONFIG_NAME value
- fbfc9a222c7f spmd_smc_handler() add s/ns state to SMC traces
- ca65081b9cdc feat(sptool): add dependency to SP image
- b1e1b46a0680 fix(qemu): restore code to added needed psci nodes
For OP-TEE I've also added a few patches, available at
https://github.com/jenswi-linaro/optee_os/tree/qemu_sel2:
- 1057def23777 plat-vexpress: sel2 spmc: update for hafnium
- f18a54ed3524 core: ffa: use hvc instead of smc with S-EL2
- d18bbc92f7c1 core: mobj_ffa_add_pages_at() trust addresses from SPMC
There's also one patch for QEMU on top of v7.0.0, available at:
https://github.com/jenswi-linaro/qemu/tree/qemu_sel2
- 0c1e39672dcb Read PS bits from VTCR_EL2
The QEMU problem is fixed in v.7.1.0, but I can't get that version of
QEMU to work with TF-A. I guess it's because of yet another new CPU
feature since I'm running with "-cpu max".
I'll try to upstream the Hafnium and TF-A patches that make sense on
their own.
What's the plan with the interrupt controller?
How will OP-TEE be able to handle secure interrupts?
The hafnium git pulls in a few git submodules and even the source code
for a Linux kernel.
I guess this is useful in your internal CI setup, but when used
isolated as in my setup it makes no sense at all.
It would also be nice to be able to build with an external toolchain.
I hope this is a temporary situation, I don't see why Hafnium should
be pickier about toolchain than for instance TF-A.
Speaking of building, I haven't been able to figure out how to build
only for the QEMU variant I need so right now I'm building for
everything and that takes a bit longer than necessary.
I'm going to maintain the setup above as long as it's relevant to me. I may
add more patches on the branches or even rebase as needed. So if anyone is
using this, keep in mind that my branches may change without warning.
Thanks,
Jens
Hi,
Thanks for your efforts for this query. I have tried your solution. Although the Hafnium code could be built successfully, it does not work when the boot flow run into the entry of the Hafnium.
I tried to debug the error with trace32 tool. And I confirmed that the Hafnium code has been loaded over 34G of free address space. However, when PC register point to the start address of Hafnium, the code data is displayed as udf #0x0.
It may be a page table problem, and we are still locating the problem reason.
Regards,
Mei, Jianqiang.
------------------------------------------------------------------
发件人：Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
发送时间：2022年12月21日(星期三) 18:02
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>; 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 黄明(连一) <hm281385(a)alibaba-inc.com>
主 题：[Hafnium] Re: Hafnium load-address
Hi,
Sorry for the delay in replying to this query.
After few trials I ended with this possible solution:
https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 <https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 >
This should permit loading the Hafnium image to an address up to 48 bits (and by updating project/reference/BUILD.gn origin_address per your suggestion).
It is possible to further extend beyond 48 bits (e.g. 52 bits which is the maximum permitted VA using FEAT_LPA2).
Let me know if this works for you, then I can proceed further with polishing this change.
Regards,
Olivier.
________________________________
From: 梅建强(禹夜) via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 11 December 2022 09:58
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>; 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 黄明(连一) <hm281385(a)alibaba-inc.com>
Subject: [Hafnium] Hafnium load-address
Hello, expert
Firstly, thanks for your answer some time ago. That's useful to our project.We combined the Hafnium (run at S-EL2) and the optee (run at S-EL1), then modified the configuration to adapt to our own environment. Now we can start the os normally.
Now we want to optimize a technical point, which is to load and run hafnium over 34G of free address space. This requires hafnium to support 64-bit load-address.
I tried to modify the configuration in BUILD.gn as follows:
origin_address = "0x880000000"
And I encountered the build error:
FAILED: secure_aem_v8a_fvp_clang/hafnium.elf ld.lld -pie --gc-sections --defsym=ORIGIN_ADDRESS=0x880000000 -O2 --icf=all --fatal-warnings --color-diagnostics -T /home/yuye/hafnium/Hafnium/build/image/image.ld -o secure_aem_v8a_fvp_clang/hafnium.elf --start-group @secure_aem_v8a_fvp_clang/hafnium.elf.rsp --end-group ld.lld: error: secure_aem_v8a_fvp_clang/obj/src/arch/aarch64/entry.entry.o:(.init.entry+0xC8): relocation R_AARCH64_ABS32 out of range: 36507222016 is not in [-2147483648, 4294967295]; references ORIGIN_ADDRESS >>> defined in -defsym:1 ninja: build stopped: subcommand failed. Makefile:70: recipe for target 'all' failed make: *** [all] Error 1 + cp out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ../../V2/Ali_M1_Public/Nonosi/Platform/Ali/M1/Bin/ cp: cannot stat 'out/reference/secure_aem_v8a_fvp_clang/hafnium.bin': No such file or directory
In this case, it seems that the new address causes some relative addressing to reach too far. Is there any solutions or supports for this issue currently in the community?
Regards,
Mei, Jianqiang.
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hello, expert
Firstly, thanks for your answer some time ago. That's useful to our project.We combined the Hafnium (run at S-EL2) and the optee (run at S-EL1), then modified the configuration to adapt to our own environment. Now we can start the os normally.
Now we want to optimize a technical point, which is to load and run hafnium over 34G of free address space. This requires hafnium to support 64-bit load-address.
I tried to modify the configuration in BUILD.gn as follows:
origin_address = "0x880000000"
And I encountered the build error:
FAILED: secure_aem_v8a_fvp_clang/hafnium.elf ld.lld -pie --gc-sections --defsym=ORIGIN_ADDRESS=0x880000000 -O2 --icf=all --fatal-warnings --color-diagnostics -T /home/yuye/hafnium/Hafnium/build/image/image.ld -o secure_aem_v8a_fvp_clang/hafnium.elf --start-group @secure_aem_v8a_fvp_clang/hafnium.elf.rsp --end-group ld.lld: error: secure_aem_v8a_fvp_clang/obj/src/arch/aarch64/entry.entry.o:(.init.entry+0xC8): relocation R_AARCH64_ABS32 out of range: 36507222016 is not in [-2147483648, 4294967295]; references ORIGIN_ADDRESS >>> defined in -defsym:1 ninja: build stopped: subcommand failed. Makefile:70: recipe for target 'all' failed make: *** [all] Error 1 + cp out/reference/secure_aem_v8a_fvp_clang/hafnium.bin ../../V2/Ali_M1_Public/Nonosi/Platform/Ali/M1/Bin/ cp: cannot stat 'out/reference/secure_aem_v8a_fvp_clang/hafnium.bin': No such file or directory
In this case, it seems that the new address causes some relative addressing to reach too far. Is there any solutions or supports for this issue currently in the community?
Regards,
Mei, Jianqiang.
Hi,
> Currently we bypass smmu and has finished boot process, later we will discuss whether there is a scenario for using smmu in the project.
That's great news!
> Yes, we use it on the server and need to support LPA.
Just as a matter of clarifying, Hafnium treats FEAT_LPA by restricting the physical address space to 48 bits:
https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/m…
The reason is that with FEAT_LPA, only the 16KB/64KB translation granules are supported, and Hafnium only supports the 4KB translation granule.
It means the normal world shall not attempt providing physical addresses (e.g. in memory sharing operations to the SPMC) where the PA uses an address size greater than 48 bits.
Is this a reasonable assumption when integrating in your system?
Secondly we were reported a slight issue when applying this restriction by Jens @ Linaro.
I appreciate the change below might be submitted shortly for upstream.
https://github.com/jenswi-linaro/hafnium/commit/659c79d5eacf32e8f5fcb1a6403…
Regards,
Olivier.
From: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Sent: 06 December 2022 10:49
To: Olivier Deprez <Olivier.Deprez(a)arm.com>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Cc: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
Subject: 回复：回复：SMMU Configuration
Hi Olivier,
Thank you very much for your reply and suggestion.
Currently we bypass smmu and has finished boot process, later we will discuss whether there is a scenario for using smmu in the project.
Other question are you relying on FEAT_LPA/FEAT_LPA2?
---Yes, we use it on the server and need to support LPA.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年12月1日(星期四) 17:12
收件人：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
主 题：Re: 回复：SMMU Configuration
Hi,
> Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24，
Thanks for confirming.
> Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage?
No, as observed here:
https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/a…
But this is an interesting input to us, as we've validated the driver with a low STEs count and this only required a linear table.
So this is an improvement we may think about.
> hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a
> yes，i have set memory range security attributes by the TZC controller.
Alright.
Before going further, do you confirm you have use cases requiring this SMMU driver/component?
In particular do you have devices upstream to this SMMU generating transactions through a secure stream ID?
e.g. this can be an SPI peripheral only accessible from the secure world?
What I mean here is there is no point in including the driver (and consuming a lot of resources) if there is no use case to fulfill eventually?
If there are, then yes, we can go ahead and find solutions to enable such use cases.
For the sake of experiment, did you attempt increasing the number of heap pages beyond 262144?
I don't have a clear idea of the memory requirements for your platform. In the reference code, the default is 180 heap pages, but I assume this can increase a lot depending on the memory size, secure partitions S2 mappings etc.
Other question are you relying on FEAT_LPA/FEAT_LPA2?
Regards,
Olivier.
From: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Sent: 01 December 2022 04:00
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
Subject: 回复：SMMU Configuration
Hi，
1、Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24， Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage？
2、 hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a5c
3、 yes，i have set memory range security attributes by the TZC controller.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年11月30日(星期三) 17:17
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; Olivier Deprez <Olivier.Deprez(a)arm.com>
抄 送：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: SMMU Configuration
Hi,
Another out of curiosity question about:
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
Hafnium as SPMC in the secure side must allocate data structures only from secure memory.
Do you confirm this memory range is marked secure by the TZASC controller?
Regards,
Olivier.
From: Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 30 November 2022 10:13
To: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
Subject: [Hafnium] Re: SMMU Configuration
Hi,
1/ About:
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24?
In which case the stream table size would grow up to 1GB.
Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation.
Can you confirm you have this change in your tree:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a06…
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 28 November 2022 10:48
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Subject: SMMU Configuration
Hi,
when I enabled SMMU by set the config file as follows：
aarch64_toolchains("secure_aem_v8a_fvp") {
cpu = "cortex-a57"
origin_address = "0x06000000"
boot_flow = "//src/boot_flow:spmc"
console = "//src/arch/aarch64/pl011"
iommu = "//src/arch/aarch64/arm_smmuv3"
gic_version = 3
gicd_base_address = "0x2e000000"
gicr_base_address = "0x2e140000"
gicr_frames = 8
heap_pages = 100000
max_cpus = 8
max_vms = 16
# branch_protection = "standard"
toolchain_args = {
plat_ffa = "//src/arch/aarch64/plat/ffa:spmc"
plat_psci = "//src/arch/aarch64/plat/psci:spmc"
plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3"
plat_prng = "//src/arch/aarch64/plat/prng:prng"
secure_world = "1"
pl011_base_address = "0x2A400000"
smmu_base_address = "0x3FC00000"
smmu_memory_size = "0x400000"
# enable_mte = "1"
plat_log_level = "LOG_LEVEL_VERBOSE"
}
}
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA
VERBOSE: SMMUv3: write to (S_)CR0
VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits
VERBOSE: SMMUv3: Total CMDQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
VERBOSE: SMMUv3: write to (S_)CMDQ_BASE
VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD
VERBOSE: SMMUv3: Total EVTQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ
VERBOSE: SMMUv3: write to (S_)EVTQ_BASE
VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
ERROR: SMMUv3: Could not allocate memory for stream table entries
ERROR: SMMUv3: Failed to initialize driver
Panic: Could not initialize IOMMUs.
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi,
> Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24，
Thanks for confirming.
> Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage?
No, as observed here:
https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/a…
But this is an interesting input to us, as we've validated the driver with a low STEs count and this only required a linear table.
So this is an improvement we may think about.
> hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a
> yes，i have set memory range security attributes by the TZC controller.
Alright.
Before going further, do you confirm you have use cases requiring this SMMU driver/component?
In particular do you have devices upstream to this SMMU generating transactions through a secure stream ID?
e.g. this can be an SPI peripheral only accessible from the secure world?
What I mean here is there is no point in including the driver (and consuming a lot of resources) if there is no use case to fulfill eventually?
If there are, then yes, we can go ahead and find solutions to enable such use cases.
For the sake of experiment, did you attempt increasing the number of heap pages beyond 262144?
I don't have a clear idea of the memory requirements for your platform. In the reference code, the default is 180 heap pages, but I assume this can increase a lot depending on the memory size, secure partitions S2 mappings etc.
Other question are you relying on FEAT_LPA/FEAT_LPA2?
Regards,
Olivier.
From: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Sent: 01 December 2022 04:00
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; 常琳(特睿) <terui.cl(a)alibaba-inc.com>
Subject: 回复：SMMU Configuration
Hi，
1、Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24， Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage？
2、 hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a5c
3、 yes，i have set memory range security attributes by the TZC controller.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年11月30日(星期三) 17:17
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>; Olivier Deprez <Olivier.Deprez(a)arm.com>
抄 送：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: SMMU Configuration
Hi,
Another out of curiosity question about:
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
Hafnium as SPMC in the secure side must allocate data structures only from secure memory.
Do you confirm this memory range is marked secure by the TZASC controller?
Regards,
Olivier.
From: Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 30 November 2022 10:13
To: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
Subject: [Hafnium] Re: SMMU Configuration
Hi,
1/ About:
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24?
In which case the stream table size would grow up to 1GB.
Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation.
Can you confirm you have this change in your tree:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a06…
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 28 November 2022 10:48
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Subject: SMMU Configuration
Hi,
when I enabled SMMU by set the config file as follows：
aarch64_toolchains("secure_aem_v8a_fvp") {
cpu = "cortex-a57"
origin_address = "0x06000000"
boot_flow = "//src/boot_flow:spmc"
console = "//src/arch/aarch64/pl011"
iommu = "//src/arch/aarch64/arm_smmuv3"
gic_version = 3
gicd_base_address = "0x2e000000"
gicr_base_address = "0x2e140000"
gicr_frames = 8
heap_pages = 100000
max_cpus = 8
max_vms = 16
# branch_protection = "standard"
toolchain_args = {
plat_ffa = "//src/arch/aarch64/plat/ffa:spmc"
plat_psci = "//src/arch/aarch64/plat/psci:spmc"
plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3"
plat_prng = "//src/arch/aarch64/plat/prng:prng"
secure_world = "1"
pl011_base_address = "0x2A400000"
smmu_base_address = "0x3FC00000"
smmu_memory_size = "0x400000"
# enable_mte = "1"
plat_log_level = "LOG_LEVEL_VERBOSE"
}
}
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA
VERBOSE: SMMUv3: write to (S_)CR0
VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits
VERBOSE: SMMUv3: Total CMDQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
VERBOSE: SMMUv3: write to (S_)CMDQ_BASE
VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD
VERBOSE: SMMUv3: Total EVTQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ
VERBOSE: SMMUv3: write to (S_)EVTQ_BASE
VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
ERROR: SMMUv3: Could not allocate memory for stream table entries
ERROR: SMMUv3: Failed to initialize driver
Panic: Could not initialize IOMMUs.
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi,
1/ About:
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24?
In which case the stream table size would grow up to 1GB.
Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation.
Can you confirm you have this change in your tree:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a06…
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 28 November 2022 10:48
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Subject: SMMU Configuration
Hi,
when I enabled SMMU by set the config file as follows：
aarch64_toolchains("secure_aem_v8a_fvp") {
cpu = "cortex-a57"
origin_address = "0x06000000"
boot_flow = "//src/boot_flow:spmc"
console = "//src/arch/aarch64/pl011"
iommu = "//src/arch/aarch64/arm_smmuv3"
gic_version = 3
gicd_base_address = "0x2e000000"
gicr_base_address = "0x2e140000"
gicr_frames = 8
heap_pages = 100000
max_cpus = 8
max_vms = 16
# branch_protection = "standard"
toolchain_args = {
plat_ffa = "//src/arch/aarch64/plat/ffa:spmc"
plat_psci = "//src/arch/aarch64/plat/psci:spmc"
plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3"
plat_prng = "//src/arch/aarch64/plat/prng:prng"
secure_world = "1"
pl011_base_address = "0x2A400000"
smmu_base_address = "0x3FC00000"
smmu_memory_size = "0x400000"
# enable_mte = "1"
plat_log_level = "LOG_LEVEL_VERBOSE"
}
}
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA
VERBOSE: SMMUv3: write to (S_)CR0
VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits
VERBOSE: SMMUv3: Total CMDQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
VERBOSE: SMMUv3: write to (S_)CMDQ_BASE
VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD
VERBOSE: SMMUv3: Total EVTQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ
VERBOSE: SMMUv3: write to (S_)EVTQ_BASE
VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
ERROR: SMMUv3: Could not allocate memory for stream table entries
ERROR: SMMUv3: Failed to initialize driver
Panic: Could not initialize IOMMUs.
