From:
Nagal, Amit <amit.nagal@amd.com>
Date: Monday, 16 December 2024 at 09:02
To: Gyorgy Szing <Gyorgy.Szing@arm.com>, Mark Dykes <Mark.Dykes@arm.com>, Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com>,
Olivier Deprez <Olivier.Deprez@arm.com>,
tf-a-tests@lists.trustedfirmware.org <tf-a-tests@lists.trustedfirmware.org>, Joanna Farley <Joanna.Farley@arm.com>,
tf-a@lists.trustedfirmware.org <tf-a@lists.trustedfirmware.org>
Cc: Jain, Ronak <ronak.jain@amd.com>, Simek, Michal <michal.simek@amd.com>
Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi ,
I am referring to the code coverage as below :
https://ci-builds.trustedfirmware.org/static-files/htxSEy4sEOQXtuGetFtzHqAmpr9kguARVj1zPKr9ahIxNzM0MzM5MTcwMjg5Ojk6YW5vbnltb3VzOmpvYi90Zi1hLWNpLWNvdmVyYWdlLWdhdGV3YXkvMTIxL2FydGlmYWN0/merge/outdir/lcov/index.html
I see a lot of files from non plat code getting covered with code coverage .
can somebody help me with further information :
-
Where are cputests related unit test scripts hosted for non plat files like bl31_main.c ?
-
can we use the unit test & code coverage infra for other platforms say AMD-Xilinx?
-
How is the code coverage achieved for assembly .S files like bl31_entrypoint.S ?
Regards
Amit
Hi Everyone,
I see very limited number of unit tests cpp files hosted at
https://git.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/refs/heads/main/tests/
Do we have more unit tests implemented for TF-A and available publicly anywhere ?
Regards
Amit
Hi Amit,
“can this be utilized to perform unit testing for platforms other than fvp”
Unit-tests executables are running on the “build host” which often is the X86 based PC of the developer (or a CI worker). UT is favoring low cost and sacrificing some level of test correctness. It assumes
the compiled C code is fully portable and works the same on all platforms. Moreover, it focuses on C function correctness, and checks if a specific function works as designed. Verifying higher level operation is out of scope (e.g. function implements any feature
correctly or is behaving correctly when called by other functions in the system). In turn the complexity of embedded development is removed (managing the target, updating the firmware, using a JTAG interface for debugging, etc…), and the resource constraints
of the embedded target is not present.
The code coverage executed on the fvp platform is a system test, it exercises the product on an emulated target. It verifies the correctness of the code as it executes like it will in a real-world scenario. This is a higher value test, but it is much more expensive
in terms of time and complexity than UT.
“Is c-picker tool available inside arm still as of now ?”
No, it is public, although it is a bit hidden. It lives on the c-picker branch to the Trusted Services repository (see [1) Similarly, Firmware Test Builder lives on the fwtb branch (see 2).
This branch is independent of TS code and contains only the two tools. We have plans to move each to dedicated repositories, but I don’t know when I will have the time to do this.
(FWTB and c-picker was developed by my team (TS) and is used and maintained by both projects. Before the TS project was started, we were working on TF-A. I can help with the tools, but I am not familiar to
TF-A specifics like, how the tools are integrated to TF-A or which parts of TF-A are covered by UT cases.)
1:
https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/refs/heads/topics/c-picker
2:
https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/refs/heads/topics/fwtb
/George
Hi Mark,
From unit test prospective , I saw this document
https://www.trustedfirmware.org/docs/TF-A-UnitLevelTesting.pdf.
can this be utilized to perform unit testing for platforms other than fvp ?
Is c-picker tool available inside arm still as of now ?
Regards
Amit
From: Mark Dykes <Mark.Dykes@arm.com>
Sent: Tuesday, October 29, 2024 9:20 PM
To: Nagal, Amit <amit.nagal@amd.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.com>;
tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.com>
Cc: Jain, Ronak <ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.com>
Subject: Re: Query regarding the test coverage thru tf-a-tests
Amit,
It appears I was incorrect in getting coverage outside of the flows we have internally. I apologize for the confusion...
Mark
________________________________
From: Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.com>>
Sent: Friday, October 25, 2024 12:07 AM
To: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com>>;
Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com>>;
tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org>>;
Joanna Farley <Joanna.Farley@arm.com<mailto:Joanna.Farley@arm.com>>
Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.com>>
Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi @Mark Dykes<mailto:Mark.Dykes@arm.com> ,
Can you please share insights about how to get coverage information using manual tests run for FVP platform .
That will help us to atleast get the coverage information for non platform code ( we are interested for bl31) .
Regards
Amit