Hi,
Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24，
Thanks for confirming.
Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage?
No, as observed here: https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/ar... But this is an interesting input to us, as we've validated the driver with a low STEs count and this only required a linear table. So this is an improvement we may think about.
hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a yes，i have set memory range security attributes by the TZC controller.
Alright.
Before going further, do you confirm you have use cases requiring this SMMU driver/component? In particular do you have devices upstream to this SMMU generating transactions through a secure stream ID? e.g. this can be an SPI peripheral only accessible from the secure world? What I mean here is there is no point in including the driver (and consuming a lot of resources) if there is no use case to fulfill eventually? If there are, then yes, we can go ahead and find solutions to enable such use cases.
For the sake of experiment, did you attempt increasing the number of heap pages beyond 262144? I don't have a clear idea of the memory requirements for your platform. In the reference code, the default is 180 heap pages, but I assume this can increase a lot depending on the memory size, secure partitions S2 mappings etc.
Other question are you relying on FEAT_LPA/FEAT_LPA2?
Regards, Olivier.
From: 赵哲(为哲) weizhe.zz@alibaba-inc.com Sent: 01 December 2022 04:00 To: Olivier Deprez Olivier.Deprez@arm.com Cc: hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org; 梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com; 常琳(特睿) terui.cl@alibaba-inc.com Subject: 回复：SMMU Configuration Hi， 1、Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24， Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage？ 2、 hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a5c 3、 yes，i have set memory range security attributes by the TZC controller.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.
------------------------------------------------------------------ 发件人：Olivier Deprez Olivier.Deprez@arm.com 发送时间：2022年11月30日(星期三) 17:17 收件人：梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com; Olivier Deprez Olivier.Deprez@arm.com 抄 送：赵哲(为哲) weizhe.zz@alibaba-inc.com; hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org 主 题：Re: SMMU Configuration
Hi,
Another out of curiosity question about: VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
Hafnium as SPMC in the secure side must allocate data structures only from secure memory. Do you confirm this memory range is marked secure by the TZASC controller?
Regards, Olivier.
From: Olivier Deprez via Hafnium hafnium@lists.trustedfirmware.org Sent: 30 November 2022 10:13 To: 梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com Cc: 赵哲(为哲) weizhe.zz@alibaba-inc.com; hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org Subject: [Hafnium] Re: SMMU Configuration Hi,
1/ About: VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24? In which case the stream table size would grow up to 1GB. Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation. Can you confirm you have this change in your tree: https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a061...
Thanks, Olivier.
________________________________ From: 梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com Sent: 28 November 2022 10:48 To: Olivier Deprez Olivier.Deprez@arm.com Cc: 赵哲(为哲) weizhe.zz@alibaba-inc.com Subject: SMMU Configuration
Hi, when I enabled SMMU by set the config file as follows： aarch64_toolchains("secure_aem_v8a_fvp") { cpu = "cortex-a57" origin_address = "0x06000000" boot_flow = "//src/boot_flow:spmc" console = "//src/arch/aarch64/pl011" iommu = "//src/arch/aarch64/arm_smmuv3" gic_version = 3 gicd_base_address = "0x2e000000" gicr_base_address = "0x2e140000" gicr_frames = 8 heap_pages = 100000 max_cpus = 8 max_vms = 16 # branch_protection = "standard" toolchain_args = { plat_ffa = "//src/arch/aarch64/plat/ffa:spmc" plat_psci = "//src/arch/aarch64/plat/psci:spmc" plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3" plat_prng = "//src/arch/aarch64/plat/prng:prng" secure_world = "1" pl011_base_address = "0x2A400000" smmu_base_address = "0x3FC00000" smmu_memory_size = "0x400000" # enable_mte = "1" plat_log_level = "LOG_LEVEL_VERBOSE" } }
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA VERBOSE: SMMUv3: write to (S_)CR0 VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits VERBOSE: SMMUv3: Total CMDQ entries: 524288 VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ VERBOSE: SMMUv3: write to (S_)CMDQ_BASE VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD VERBOSE: SMMUv3: Total EVTQ entries: 524288 VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ VERBOSE: SMMUv3: write to (S_)EVTQ_BASE VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS VERBOSE: SMMUv3 Total StreamTable entries: 16777216 ERROR: SMMUv3: Could not allocate memory for stream table entries ERROR: SMMUv3: Failed to initialize driver Panic: Could not initialize IOMMUs.
Hi Olivier, Thank you very much for your reply and suggestion. Currently we bypass smmu and has finished boot process, later we will discuss whether there is a scenario for using smmu in the project. Other question are you relying on FEAT_LPA/FEAT_LPA2? ---Yes, we use it on the server and need to support LPA. --------------------------------------------------------------------------------------------------------------------------------------------------------------- 本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it. ------------------------------------------------------------------ 发件人：Olivier Deprez Olivier.Deprez@arm.com 发送时间：2022年12月1日(星期四) 17:12 收件人：赵哲(为哲) weizhe.zz@alibaba-inc.com; 梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com 抄 送：hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org; 常琳(特睿) terui.cl@alibaba-inc.com 主 题：Re: 回复：SMMU Configuration Hi,
Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24，
Thanks for confirming.
Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage?
No, as observed here: https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/ar... <https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/ar... > But this is an interesting input to us, as we've validated the driver with a low STEs count and this only required a linear table. So this is an improvement we may think about.
hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a yes，i have set memory range security attributes by the TZC controller.
Alright. Before going further, do you confirm you have use cases requiring this SMMU driver/component? In particular do you have devices upstream to this SMMU generating transactions through a secure stream ID? e.g. this can be an SPI peripheral only accessible from the secure world? What I mean here is there is no point in including the driver (and consuming a lot of resources) if there is no use case to fulfill eventually? If there are, then yes, we can go ahead and find solutions to enable such use cases. For the sake of experiment, did you attempt increasing the number of heap pages beyond 262144? I don't have a clear idea of the memory requirements for your platform. In the reference code, the default is 180 heap pages, but I assume this can increase a lot depending on the memory size, secure partitions S2 mappings etc. Other question are you relying on FEAT_LPA/FEAT_LPA2? Regards, Olivier. From: 赵哲(为哲) weizhe.zz@alibaba-inc.com Sent: 01 December 2022 04:00 To: Olivier Deprez Olivier.Deprez@arm.com Cc: hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org; 梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com; 常琳(特睿) terui.cl@alibaba-inc.com Subject: 回复：SMMU Configuration Hi， 1、Since our platform uses the server's cpu, smmu does have SMMU_S_IDR1.S_SIDSIZE=24， Does the smmu driver of Hafnium support 2-level Stream Table to reduce memory usage？ 2、 hafnium commit hash dd883207ee9b31c19169adf97c918d561dcb9a5c 3、 yes，i have set memory range security attributes by the TZC controller. --------------------------------------------------------------------------------------------------------------------------------------------------------------- 本邮件及其附件含有阿里巴巴集团的商业秘密信息，仅限于发送给上面地址中列出的个人和群组，禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息，如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。 This email and its attachments contain confidential information from Alibaba Group, which is intended only for the person or entity whose address is listed above. Any use of information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it. ------------------------------------------------------------------ 发件人：Olivier Deprez Olivier.Deprez@arm.com 发送时间：2022年11月30日(星期三) 17:17 收件人：梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com; Olivier Deprez Olivier.Deprez@arm.com 抄 送：赵哲(为哲) weizhe.zz@alibaba-inc.com; hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org 主 题：Re: SMMU Configuration Hi, Another out of curiosity question about: VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ Hafnium as SPMC in the secure side must allocate data structures only from secure memory. Do you confirm this memory range is marked secure by the TZASC controller? Regards, Olivier. From: Olivier Deprez via Hafnium hafnium@lists.trustedfirmware.org Sent: 30 November 2022 10:13 To: 梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com Cc: 赵哲(为哲) weizhe.zz@alibaba-inc.com; hafnium@lists.trustedfirmware.org hafnium@lists.trustedfirmware.org Subject: [Hafnium] Re: SMMU Configuration Hi, 1/ About: VERBOSE: SMMUv3 Total StreamTable entries: 16777216 I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24? In which case the stream table size would grow up to 1GB. Can you confirm this parameter value on your platform? Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address? 2/ can you tell which commit hash you're using for hafnium? We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation. Can you confirm you have this change in your tree: https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a061... <https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a061... > Thanks, Olivier. ________________________________ From: 梅建强(禹夜) meijianqiang.mjq@alibaba-inc.com Sent: 28 November 2022 10:48 To: Olivier Deprez Olivier.Deprez@arm.com Cc: 赵哲(为哲) weizhe.zz@alibaba-inc.com Subject: SMMU Configuration Hi, when I enabled SMMU by set the config file as follows： aarch64_toolchains("secure_aem_v8a_fvp") { cpu = "cortex-a57" origin_address = "0x06000000" boot_flow = "//src/boot_flow:spmc" console = "//src/arch/aarch64/pl011" iommu = "//src/arch/aarch64/arm_smmuv3" gic_version = 3 gicd_base_address = "0x2e000000" gicr_base_address = "0x2e140000" gicr_frames = 8 heap_pages = 100000 max_cpus = 8 max_vms = 16 # branch_protection = "standard" toolchain_args = { plat_ffa = "//src/arch/aarch64/plat/ffa:spmc" plat_psci = "//src/arch/aarch64/plat/psci:spmc" plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3" plat_prng = "//src/arch/aarch64/plat/prng:prng" secure_world = "1" pl011_base_address = "0x2A400000" smmu_base_address = "0x3FC00000" smmu_memory_size = "0x400000" # enable_mte = "1" plat_log_level = "LOG_LEVEL_VERBOSE" } } I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions? VERBOSE: SMMUv3 mapped at 000000003fc00000 VERBOSE: SMMUv3: write to (S_)GBPA VERBOSE: SMMUv3: write to (S_)CR0 VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits VERBOSE: SMMUv3: Total CMDQ entries: 524288 VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ VERBOSE: SMMUv3: write to (S_)CMDQ_BASE VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD VERBOSE: SMMUv3: Total EVTQ entries: 524288 VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ VERBOSE: SMMUv3: write to (S_)EVTQ_BASE VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS VERBOSE: SMMUv3 Total StreamTable entries: 16777216 ERROR: SMMUv3: Could not allocate memory for stream table entries ERROR: SMMUv3: Failed to initialize driver Panic: Could not initialize IOMMUs.
hafnium@lists.trustedfirmware.org
-
Olivier Deprez -
赵哲(为哲)