Hi Jaehyuk,

See comments inline [OD]

Regards, Olivier.

________________________________ From: Lee, Jaehyuk via TF-A-Tests tf-a-tests@lists.trustedfirmware.org Sent: 01 April 2023 01:18 To: tf-a-tests@lists.trustedfirmware.org tf-a-tests@lists.trustedfirmware.org Subject: [Tf-a-tests] Question about SMMU testing

HI, tf-a-test community,

I have one question about the smmuv3 testing on fvp. Below is the SMMUV3 related parameters passed to fvp suggested in https://trustedfirmware-a.readthedocs.io/en/latest/components/realm-manageme... to enable 3rd world options for RME (CCA).

pci.pci_smmuv3.mmu.SMMU_ROOT_IDR0=3

pci.pci_smmuv3.mmu.SMMU_ROOT_IIDR=0x43B

pci.pci_smmuv3.mmu.root_register_page_offset=0x20000

[OD] Those parameters are mainly to permit enabling GPC on secure and non-secure SMMU transactions, when RME is implemented. This is done at boot by the minimal EL3 SMMU driver: https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/tree/drivers/arm...

What I want to try with fvp with SMMU is generating transactions from the peripheral devices through the SMMU to access some memory ranges.

I found that FVP doesn’t have any peripherals connected through the SMMU, shown in the dts file in the linux repository for FVP, except PCIe (i.e., pci: pci@40000000 in fvp-base-revc.dts)

Is there any way that I can attach any peripherals through the SMMU or generate transactions to SMMU directly to make SMMU walks the stream table, which allows the device to access memories?

[OD] AFAIK the SMMU integrated on Base FVP has a single upstream peripheral, the SMMUv3TestEngine. This is basically a DMA loopback device able to generate non-secure or secure transactions. I don't know of a capability to add other types of peripherals, in an easy way. Esp. SMMU+PCIe at first sight I would say this isn't supported with the current model. Importantly, as I see you refer to linux, are you interested by normal world/non-secure only transactions? TF-A-tests has support for testing secure transactions mainly. And no support for 'Realm' transactions just yet.

I checked the spm/cactus/cactus_tests/cactus_tests_smmuv3.c But seems that this test case is not executable on the fvp? I got the “FFA_VERSION” not supported error message. Moreover, it seems that this test just send command to emulated smmutesting engine and do memcopy operation. I cannot understand how does it work even without initializing stream table (as far as I know it is initialized with kernel driver).

[OD] You are on the right track, but I should add a couple more details: This test is the only test part of TF-A test suite functionally exercising the SMMU and the SMMUv3TestEngine. TFTF runs at NS EL2 and kick starts the SP to perform the DMA operation in the secure world. 'Cactus' runs as a SEL1 Secure Partition. The SMMUv3TestEngine is assigned to this SEL1 SP. Cactus controls the test engine through MMIO, to define address ranges for the 'copy' operation, and then start the DMA engine. Hafnium runs at SEL2, and by its SMMU driver controls the peripheral's secure S2 translations and associated PTs.

You can see test results in the CI: https://ci-builds.trustedfirmware.org/static-files/tTwKFOJCfHbMZy-xQrKsq6LsR...

Running test suite 'SMMUv3 tests' Description: Initiate stage2 translation for streams from upstream peripherals

Executing 'Check DMA command by SMMUv3TestEngine completes'

TEST COMPLETE Passed

See the logs tagged by pci_smmuv3 https://ci-builds.trustedfirmware.org/static-files/reujri7dYIf0M090c-wAipktg...

I can provide command lines, if you want to replicate this.

Sincerely Jaehyuk -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org

Hi Jaehyuk,

You are correct about SMMUv3 supporting both secure and non-secure transactions. Essentially, it behaves as if there are two independent SMMU instances (one for secure and the other for non-secure streams). However, since our goal was to test Secure Stage2 translation support in Hafnium SPMC for transactions generated by a Secure Partition, we have not done testing around generating NS transactions from the normal world. If I am not mistaken, the user frame of SMMUv3TestEngine can be configured to generate non-secure transactions by setting the attribute field appropriately[1]. There is not much documentation about programming the test engine component[4]. We used a header file that has some embedded documentation to create the test that generates secure transactions(cactus_tests_smmuv3.c) but it is not publicly available to developers outside of Arm. I have provided few links below that you might find useful.

As Olivier mentioned, I don't think there are other peripherals upstream of SMMUv3 IP in BaseAEM RevC FVP fast model [2] except SMMUv3TestEngine. This model is available to download from [3].

[1] https://developer.arm.com/documentation/107925/1121/Trace-components/SMMUv3T... [2] https://developer.arm.com/documentation/100964/1121/Base-Platform/Base-Platf... [3] https://developer.arm.com/Tools%20and%20Software/Fast%20Models [4] https://developer.arm.com/documentation/100964/1121/Base-Platform/Base-Platf...

Thanks, Madhukar

-----Original Message----- From: Lee, Jaehyuk via TF-A-Tests tf-a-tests@lists.trustedfirmware.org Sent: Monday, April 3, 2023 1:16 PM To: Olivier Deprez Olivier.Deprez@arm.com Cc: tf-a-tests@lists.trustedfirmware.org Subject: [Tf-a-tests] Re: Question about SMMU testing

Thank you so much for clarification. I find that you mentioned SMMUv3TestEngine is designed for generating "non-secure" or secure transactions but seems that cactus is designed to run at secure world to test SMMU with secure world transactions.

AFAIK, the SMMU can support S/NS transactions by having separate pairs of registers control the stream tables and configurations so that NW transactions cannot access the SW memories through SMMU.

If I want to generate transactions to the SMMU through SMMUv3TestEngine is there any examples I can take a look? If I can send read/write transactions to SMMU on behalf of NW device, not SW device, that should be enough at this moment.

Also you mentioned that there is no way to implement PCIe device connected through SMMU, then does it also mean that there is no way to add device similar to the SMMUv3TestEngine? I would like to test capabilities of SMMU if that emulation is possible in addition to currently supported read/write requests.

If you can provide command line to replicate the log, it would be really helpful.

Sincerely Jaehyuk

From: Olivier Deprez Olivier.Deprez@arm.com Date: Monday, April 3, 2023 at 10:07 AM To: Lee, Jaehyuk jaehyuk@gatech.edu Cc: tf-a-tests@lists.trustedfirmware.org tf-a-tests@lists.trustedfirmware.org Subject: Re: Question about SMMU testing Hi Jaehyuk,

See comments inline [OD]

Regards, Olivier.

________________________________ From: Lee, Jaehyuk via TF-A-Tests tf-a-tests@lists.trustedfirmware.org Sent: 01 April 2023 01:18 To: tf-a-tests@lists.trustedfirmware.org tf-a-tests@lists.trustedfirmware.org Subject: [Tf-a-tests] Question about SMMU testing

HI, tf-a-test community,

I have one question about the smmuv3 testing on fvp. Below is the SMMUV3 related parameters passed to fvp suggested in https://trustedfirmware-a.readthedocs.io/en/latest/components/realm-manageme... to enable 3rd world options for RME (CCA).

pci.pci_smmuv3.mmu.SMMU_ROOT_IDR0=3

pci.pci_smmuv3.mmu.SMMU_ROOT_IIDR=0x43B

pci.pci_smmuv3.mmu.root_register_page_offset=0x20000

[OD] Those parameters are mainly to permit enabling GPC on secure and non-secure SMMU transactions, when RME is implemented. This is done at boot by the minimal EL3 SMMU driver: https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/tree/drivers/arm...

What I want to try with fvp with SMMU is generating transactions from the peripheral devices through the SMMU to access some memory ranges.

I found that FVP doesn't have any peripherals connected through the SMMU, shown in the dts file in the linux repository for FVP, except PCIe (i.e., pci: pci@40000000 in fvp-base-revc.dts)

Is there any way that I can attach any peripherals through the SMMU or generate transactions to SMMU directly to make SMMU walks the stream table, which allows the device to access memories?

[OD] AFAIK the SMMU integrated on Base FVP has a single upstream peripheral, the SMMUv3TestEngine. This is basically a DMA loopback device able to generate non-secure or secure transactions. I don't know of a capability to add other types of peripherals, in an easy way. Esp. SMMU+PCIe at first sight I would say this isn't supported with the current model. Importantly, as I see you refer to linux, are you interested by normal world/non-secure only transactions? TF-A-tests has support for testing secure transactions mainly. And no support for 'Realm' transactions just yet.

I checked the spm/cactus/cactus_tests/cactus_tests_smmuv3.c But seems that this test case is not executable on the fvp? I got the "FFA_VERSION" not supported error message. Moreover, it seems that this test just send command to emulated smmutesting engine and do memcopy operation. I cannot understand how does it work even without initializing stream table (as far as I know it is initialized with kernel driver).

[OD] You are on the right track, but I should add a couple more details: This test is the only test part of TF-A test suite functionally exercising the SMMU and the SMMUv3TestEngine. TFTF runs at NS EL2 and kick starts the SP to perform the DMA operation in the secure world. 'Cactus' runs as a SEL1 Secure Partition. The SMMUv3TestEngine is assigned to this SEL1 SP. Cactus controls the test engine through MMIO, to define address ranges for the 'copy' operation, and then start the DMA engine. Hafnium runs at SEL2, and by its SMMU driver controls the peripheral's secure S2 translations and associated PTs.

You can see test results in the CI: https://ci-builds.trustedfirmware.org/static-files/tTwKFOJCfHbMZy-xQrKsq6LsR...

Running test suite 'SMMUv3 tests'

Description: Initiate stage2 translation for streams from upstream peripherals

Executing 'Check DMA command by SMMUv3TestEngine completes'

TEST COMPLETE Passed See the logs tagged by pci_smmuv3 https://ci-builds.trustedfirmware.org/static-files/reujri7dYIf0M090c-wAipktg...

I can provide command lines, if you want to replicate this.

Sincerely Jaehyuk -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org