Hi Senthil, As Olivier indicates the Code Coverage tooling the TF-A project has is tightly integrated to the OpenCI system we use to measure coverage for all the testing that is run through the OpenCI Jenkins Test Jobs https://ci.trustedfirmware.org/view/TF-A/job/tf-a-ci-coverage-gateway/.

However, the underpinnings are through a code coverage mechanism provided out of the coverage tool on in the QA-Tools opensource project https://gitlab.arm.com/tooling/qa-tools see https://gitlab.arm.com/tooling/qa-tools/-/blob/master/coverage-tool/readme.m...

In short it relies on the coverage test runs being run on Arm FVP (Fixed Virtual Platform) models with a particular model plugin called Model Trace Interface (MTI) https://developer.arm.com/Tools%20and%20Software/Fixed%20Virtual%20Platforms

The Code Coverage tooling User guide provides more details https://gitlab.arm.com/tooling/qa-tools/-/blob/master/coverage-tool/docs/use...

In short though trace logs generated by tests run on FVP platforms are processed by the code coverage tools along side the source code to generate the data to be handled by LCOV.

You could run your tests directly on a FVP platform on your developer machine to generate code cover information using the above information.

However, I may be wrong though but I suspect you want to run your tests or existing TF-A Tests on your own hardware platform and get code coverage information? If so to do that you would have to work out how to generate trace logs compatible to the qa-tools code coverage tooling. See the design information in the code coverage tooling user guide.

You would need to talk to the Owners of the qa-tools project if you need guidance on what is possible https://gitlab.arm.com/tooling/qa-tools/-/project_members unfortunately we in the TF-A project are more consumers/users of the qa-tools code coverage tooling.

HTH

Joanna

From: Olivier Deprez via TF-A-Tests tf-a-tests@lists.trustedfirmware.org Date: Wednesday, 28 August 2024 at 06:53 To: Mark Dykes Mark.Dykes@arm.com, tf-a-tests@lists.trustedfirmware.org tf-a-tests@lists.trustedfirmware.org, Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com Cc: Jain, Ronak ronak.jain@amd.com, Nagal, Amit amit.nagal@amd.com, Simek, Michal michal.simek@amd.com Subject: [Tf-a-tests] Re: Query regarding the test coverage thru tf-a-tests Hi Senthil,

* Is there a way to get code coverage for manual test runs?

Sorry, I'm not aware of such capability. As you say code coverage implies use of CI scripting that isn't possible to reproduce locally out of the box (afaik).

Regards, Olivier.

________________________________ From: Thangaraj, Senthil Nathan via TF-A-Tests tf-a-tests@lists.trustedfirmware.org Sent: 26 August 2024 23:58 To: Mark Dykes Mark.Dykes@arm.com; Olivier Deprez Olivier.Deprez@arm.com; tf-a-tests@lists.trustedfirmware.org tf-a-tests@lists.trustedfirmware.org Cc: Jain, Ronak ronak.jain@amd.com; Nagal, Amit amit.nagal@amd.com; Simek, Michal michal.simek@amd.com Subject: [Tf-a-tests] Re: Query regarding the test coverage thru tf-a-tests

Hi Mark and Oliver,

A gentle reminder on the below request.

Thanks, Senthil

From: Thangaraj, Senthil Nathan Sent: Tuesday, August 20, 2024 3:44 PM To: Mark Dykes Mark.Dykes@arm.com; Olivier Deprez Olivier.Deprez@arm.com; tf-a-tests@lists.trustedfirmware.org Cc: Jain, Ronak ronak.jain@amd.com; Nagal, Amit amit.nagal@amd.com; Simek, Michal michal.simek@amd.com Subject: RE: Query regarding the test coverage thru tf-a-tests

Thanks a lot Mark and Oliver for your inputs and pointers.

All of these seem to be more closely integrated with the CI/CD builds. Is there a way to get code coverage for manual test runs? Specifically, I'm looking to get coverage results after loading and executing all the tests from a developer build (e.g., using a binary like tftf.bin).

Best regards, Senthil

From: Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com> Sent: Tuesday, August 20, 2024 3:16 PM To: Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com> Subject: RE: Query regarding the test coverage thru tf-a-tests

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

All, This link is more current. The overall coverage numbers are not being reported due to an issue that is being worked on now. However you can see the individual coverage metrics if you click on a particular config where is says "success". The next page will have the link "Build Artifacts" at the top and so click that to the next where you will see the directory "trace_report". Click this and you will see a list of contents one of which should be index.html. If you click this you will see the individual coverage report for that config(test).

tf-a-ci-coverage-gateway #72 [Jenkins] (trustedfirmware.org)https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/72/

Let me know if you have questions...

Mark From: Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com> Sent: Tuesday, August 20, 2024 2:27 AM To: tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com> Subject: Re: Query regarding the test coverage thru tf-a-tests

Hi,

You can refer to this job showing code coverage results aggregation from multiple test configurations:

https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/67/

https://ci-builds.trustedfirmware.org/static-files/IK1tucCNXFdhS4AAQth8N7zWg...

I don't think there is a test level granularity. @Mark Dykesmailto:Mark.Dykes@arm.com may know better.

Regards, Olivier.

________________________________ From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org> Sent: 20 August 2024 08:40 To: tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org> Subject: [Tf-a-tests] Query regarding the test coverage thru tf-a-tests

Dear TF-A team,

I have a query regarding the TF-A test. Specifically, I would like to know if there is a method to find out the code coverage for a specific test or for all the tests in current run?

Your direction in this regard will be really appreciated.

Thank you, Senthil -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org