Hi Chris,

Thanks. The structure looks very promising.

I suppose it won’t be a ground-breaking change. Will propose a solution later.

Best regards,

Hu Ziji

From: Chris.Brand@infineon.com <Chris.Brand@infineon.com>
Sent: Thursday, August 26, 2021 1:27 AM
To: David Hu <David.Hu@arm.com>; tf-m@lists.trustedfirmware.org
Cc: nd <nd@arm.com>
Subject: RE: [RFC] Decoupling tf-m-tests from TF-M

From a very high-level perspective, it would be nice to have test/test_services/platform under which any platform-specific test services could live. More importantly, a platform-specific directory for tests under test/suites somewhere would be nice – there’s already test/suites/platform, but that is aimed at testing the platform service itself. I imagine that there could be both platform-specific parts of the platform test as well as platform-specific test suites.

The key advantage of being able to upstream platform-specific tests is for regression testing to pick up breakages from changes made elsewhere (assuming the platform itself is integrated into the TFM CI loop).

Chris

From: David Hu <David.Hu@arm.com>
Sent: Wednesday, August 25, 2021 1:18 AM
To: Brand Chris (CSCA CSS ICW SW PSW 1) <Chris.Brand@infineon.com>; tf-m@lists.trustedfirmware.org
Cc: nd <nd@arm.com>
Subject: RE: [RFC] Decoupling tf-m-tests from TF-M

Caution: This e-mail originated outside Infineon Technologies. Do not click on links or open attachments unless you validate it is safe.

Hi Chris,

It is an excellent suggestion.

Out-of-tree Secure Partition build can help integrate secure test service.

Non-secure tests are a bit limited due to current tf-m-tests framework right now.

Do you prefer to run platform-specific tests alone or still integrate platform-specific tests into TF-M regression tests?

Best regards,

Hu Ziji

From: TF-M <tf-m-bounces@lists.trustedfirmware.org> On Behalf Of chris.brand--- via TF-M
Sent: Tuesday, August 24, 2021 11:46 PM
To: tf-m@lists.trustedfirmware.org
Subject: Re: [TF-M] [RFC] Decoupling tf-m-tests from TF-M

Just wondering if any though has been given to supporting platform-specific tests?

Chris

From: TF-M <tf-m-bounces@lists.trustedfirmware.org> On Behalf Of David Hu via TF-M
Sent: Tuesday, August 24, 2021 3:21 AM
To: Andrej Butok <andrey.butok@nxp.com>
Cc: tf-m@lists.trustedfirmware.org
Subject: Re: [TF-M] [RFC] Decoupling tf-m-tests from TF-M

Caution: This e-mail originated outside Infineon Technologies. Do not click on links or open attachments unless you validate it is safe.

Hi Andrej,

Thanks for the suggestion. Sure. I will track it in the backlog.

Currently Jianliang and I are more focusing on the structure level enhancement. But definitely later we will take more effort in the detailed optimizations.

Please let us know any time if any other potential issue shall be optimized.

Best regards,

Hu Ziji

From: Andrej Butok <andrey.butok@nxp.com>
Sent: Tuesday, August 24, 2021 6:15 PM
To: David Hu <David.Hu@arm.com>
Cc: tf-m@lists.trustedfirmware.org
Subject: RE: [RFC] Decoupling tf-m-tests from TF-M

Hi Hu Zij,

Thank you for adding possibility to select test cases flexibly.

Also, there are about 10 “test” services/partitions in addition to the core PSA ones.

But every instance allocates own resources, which can be shared.

Guess, merging these 10 test services, which have a common structure, can save some memory.

Thank you,

Andrej

From: David Hu <David.Hu@arm.com>
Sent: Tuesday, August 24, 2021 11:54 AM
To: Andrej Butok <andrey.butok@nxp.com>
Cc: tf-m@lists.trustedfirmware.org
Subject: RE: [RFC] Decoupling tf-m-tests from TF-M

Hi Andrej,

Sorry for the trouble.

It does be an issue when TF-M features and test cases are growing faster.

So now TF-M support to select a single test case or a subset of test cases to build and run. If running all the tests together costs too much memory, you can select some test cases or just a single one in one time.

It is also helpful when you focus on a specific test in debug or development.

We are also considering other additional mechanisms to select test case flexibly.

Regarding “merging existing ones”, do you mean that some test cases shall be disabled by default or combining the similar test cases? May I ask for some examples?

Best regards,

Hu Ziji

From: Andrej Butok <andrey.butok@nxp.com>
Sent: Tuesday, August 24, 2021 5:44 PM
To: David Hu <David.Hu@arm.com>
Cc: tf-m@lists.trustedfirmware.org
Subject: RE: [RFC] Decoupling tf-m-tests from TF-M

Hi Hu Ziji,

BTW:

The number of the testing partitions and services is growing consuming memory resources.

So, we have to disable some tests for our memory constrained devices.

Please think about minimizing number of “testing” partitions/services, by merging existing ones, when it is possible.

Thank you,

Andrej Butok

SW Tech Lead

Edge Processing

NXP Semiconductors

From: TF-M <tf-m-bounces@lists.trustedfirmware.org> On Behalf Of David Hu via TF-M
Sent: Tuesday, August 24, 2021 11:33 AM
To: tf-m@lists.trustedfirmware.org
Cc: nd <nd@arm.com>
Subject: [TF-M] [RFC] Decoupling tf-m-tests from TF-M

Hi all,

As you may know, Jianliang and I are working to better decouple tf-m-tests from trusted-firmware-m repo.

The purpose of the decoupling enhancement includes:

• Making it more easier to integrate TF-M and port tf-m-tests
• Making it more easier to develop TF-M tests, to minimize the changes to TF-M source code or build system.
• Making it more flexible to re-structure tf-m-tests and minimize the impact to TF-M

Previously Jianliang has decouple test case control and enable users to select single NS/S regression test case in build and test.

Currently we are focusing on decoupling tf-m-tests specific config setting from TF-M.

So far we have proposed the following major changes:

• Decouple tf-m-tests specific config setting from trusted-firmware-m.

[TF-M patch][tf-m-tests patch]

• Move tf-m-tests specific configs to tf-m-tests repo from trusted-firmware-m

[TF-M patch][tf-m-tests patch]

More patch sets for decoupling are under review as well.

• Decouple tf-m-tests secure log from non-secure log. Switch tf-m-tests secure log to TF-M SP log.

[TF-M patch][tf-m-tests patch]

• Trigger secure regression tests in TF-M SPE in IPC model, to simplify multi-core development/tests

[TF-M patch][tf-m-tests patch]

I’d appreciate it if you can take a look at the patch sets above. Any suggestion or comment is welcome.

If you have any specific requirement or suggestion of tf-m-tests enhancement, please feel free to contact Jianliang and me.

Thanks in advance.

Best regards,

Hu Ziji