Re: Query regarding the test coverage thru tf-a-tests
by Thangaraj, Senthil Nathan
Thank you Joanna and Mark, for your insights and response.
I wasn't aware that these tests are running on a Virtual Platform and that's how the coverage is being measured. I'll look into this further, decide how we want to integrate it into our project, and get back to you.
I appreciate your help and support here.
Best regards,
Senthil
From: Mark Dykes <Mark.Dykes(a)arm.com>
Sent: Wednesday, August 28, 2024 9:03 AM
To: Olivier Deprez <Olivier.Deprez(a)arm.com>; tf-a-tests(a)lists.trustedfirmware.org; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj(a)amd.com>
Cc: Jain, Ronak <ronak.jain(a)amd.com>; Nagal, Amit <amit.nagal(a)amd.com>; Simek, Michal <michal.simek(a)amd.com>
Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Senthil,
I do have a way to get coverage for manual test runs using FVP if that helps. I can assist if needed....
Mark
From: Olivier Deprez <Olivier.Deprez(a)arm.com<mailto:Olivier.Deprez@arm.com>>
Sent: Wednesday, August 28, 2024 12:53 AM
To: Mark Dykes <Mark.Dykes(a)arm.com<mailto:Mark.Dykes@arm.com>>; tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj(a)amd.com<mailto:SenthilNathan.Thangaraj@amd.com>>
Cc: Jain, Ronak <ronak.jain(a)amd.com<mailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal(a)amd.com<mailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek(a)amd.com<mailto:michal.simek@amd.com>>
Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Senthil,
* Is there a way to get code coverage for manual test runs?
Sorry, I'm not aware of such capability.
As you say code coverage implies use of CI scripting that isn't possible to reproduce locally out of the box (afaik).
Regards,
Olivier.
________________________________
From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org>>
Sent: 26 August 2024 23:58
To: Mark Dykes <Mark.Dykes(a)arm.com<mailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez(a)arm.com<mailto:Olivier.Deprez@arm.com>>; tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org>>
Cc: Jain, Ronak <ronak.jain(a)amd.com<mailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal(a)amd.com<mailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek(a)amd.com<mailto:michal.simek@amd.com>>
Subject: [Tf-a-tests] Re: Query regarding the test coverage thru tf-a-tests
Hi Mark and Oliver,
A gentle reminder on the below request.
Thanks,
Senthil
From: Thangaraj, Senthil Nathan
Sent: Tuesday, August 20, 2024 3:44 PM
To: Mark Dykes <Mark.Dykes(a)arm.com<mailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez(a)arm.com<mailto:Olivier.Deprez@arm.com>>; tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org>
Cc: Jain, Ronak <ronak.jain(a)amd.com<mailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal(a)amd.com<mailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek(a)amd.com<mailto:michal.simek@amd.com>>
Subject: RE: Query regarding the test coverage thru tf-a-tests
Thanks a lot Mark and Oliver for your inputs and pointers.
All of these seem to be more closely integrated with the CI/CD builds. Is there a way to get code coverage for manual test runs? Specifically, I'm looking to get coverage results after loading and executing all the tests from a developer build (e.g., using a binary like tftf.bin).
Best regards,
Senthil
From: Mark Dykes <Mark.Dykes(a)arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>>
Sent: Tuesday, August 20, 2024 3:16 PM
To: Olivier Deprez <Olivier.Deprez(a)arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>>; tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj(a)amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>>
Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
All,
This link is more current. The overall coverage numbers are not being reported due to an issue that is being worked on now. However you can see the individual coverage metrics if you click on a particular config where is says "success". The next page will have the link "Build Artifacts" at the top and so click that to the next where you will see the directory "trace_report". Click this and you will see a list of contents one of which should be index.html. If you click this you will see the individual coverage report for that config(test).
tf-a-ci-coverage-gateway #72 [Jenkins] (trustedfirmware.org)<https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/72/>
Let me know if you have questions...
Mark
From: Olivier Deprez <Olivier.Deprez(a)arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>>
Sent: Tuesday, August 20, 2024 2:27 AM
To: tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj(a)amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>>; Mark Dykes <Mark.Dykes(a)arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>>
Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi,
You can refer to this job showing code coverage results aggregation from multiple test configurations:
https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/67/
https://ci-builds.trustedfirmware.org/static-files/IK1tucCNXFdhS4AAQth8N7zW…
I don't think there is a test level granularity. @Mark Dykes<mailto:Mark.Dykes@arm.com> may know better.
Regards,
Olivier.
________________________________
From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>>
Sent: 20 August 2024 08:40
To: tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> <tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>>
Subject: [Tf-a-tests] Query regarding the test coverage thru tf-a-tests
Dear TF-A team,
I have a query regarding the TF-A test. Specifically, I would like to know if there is a method to find out the code coverage for a specific test or for all the tests in current run?
Your direction in this regard will be really appreciated.
Thank you,
Senthil
--
TF-A-Tests mailing list -- tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>
To unsubscribe send an email to tf-a-tests-leave(a)lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org>>
--
TF-A-Tests mailing list -- tf-a-tests(a)lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org>
To unsubscribe send an email to tf-a-tests-leave(a)lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org>
