Hi,

Any thoughts/feedback on separating build and post-build dependencies?

Thanks, Dev

On 23/10/2019, 16:04, "Devaraj Ranganna" Devaraj.Ranganna@arm.com wrote:

Hi,

Thanks for your inputs.

The original intention of the script was to not to modify the origin of the information i.e rst file as Minos quoted and also to avoid duplication of information. Considering all the new information, I think we should try to separate build and post-build dependencies (ex: srec_cat). This separation will allow us to handle all the build dependencies using "cmake" and then we can explore the possible option on how to handle post-build dependencies.

@Minos, I believe all sw dependencies defined in https://ci.trustedfirmware.org/job/tf-m-build-test-nightly/lastSuccessfulBui... are already validated in cmake. Adding other library dependencies into it will make it complete. This way we don’t need mention library dependency in document anymore and still achieve single source of information.

@George, I think first 3 question can be answered if we decide to go ahead with handling build dependencies with cmake.

Thanks, Dev

On 21/10/2019, 15:56, "TF-M on behalf of Gyorgy Szing via TF-M" <tf-m-bounces@lists.trustedfirmware.org on behalf of tf-m@lists.trustedfirmware.org> wrote:

Hi,

We need to keep an eye on some factors which the current prototype ignores. Some that come to my mind:

1. which component shall be the owner of sw dependency info? (Documentation, cmake, something else?). 2. how does the solution scale? (i.e. be able to handle platform specific dependencies). 3. how we handle build configuration specific dependencies? (i.e. if I don't build a service then some dependency is not needed). 4. As Minos mentioned some environment verification is already done by CMake. Is it worth to extract all dependency info (tooling + SW) into a dedicated place?

/George

-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Minos Galanakis via TF-M Sent: 21 October 2019 16:47 To: tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

Hi,

Making CMAKE able to output the dependencies it is expecting is the quickest path, but it is only able to resolve a subset of the requirements captured by the documenation. .https://ci.trustedfirmware.org/job/tf-m-build-test-nightly/lastSuccessfulBuild/artifact/build-docs/tf-m_documents/install/doc/user_guide/html/docs/user_guides/tfm_sw_requirement.html For example the cmake version itself, or tools like srec_cat used to produce MUSCA binaries. There are certain dependencies that are provided by other means ( package managers ) .

So while I agree that it looks quite odd, it is hierarchically the origin of the information we are trying to capture.

Minos ________________________________ From: TF-M tf-m-bounces@lists.trustedfirmware.org on behalf of Kumar Gala via TF-M tf-m@lists.trustedfirmware.org Sent: 21 October 2019 15:25 To: Devaraj Ranganna Devaraj.Ranganna@arm.com Cc: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

This feels backwards, parsing rst to determine build steps, dependancies doesn’t feel correct. I think it would be better to codify the build steps in scripts of CMake, etc than reference those in the .rst instead.

- k

> On Oct 21, 2019, at 9:03 AM, Devaraj Ranganna via TF-M tf-m@lists.trustedfirmware.org wrote: > > Hi, > > Currently tf-m dependencies are only listed in documentation file (docs/user_guides/tfm_build_instruction.rst) and there is no option to programmatically retrieve it. In order to integrate tf-m into non-secure side RTOSes, it is very essential to be able build tf-m automatically without manual intervention (may be as part of CI). > > The tf-m CI build system uses fixed values for dependencies and it is the responsibility of developers who update the dependencies to update the documentation and the CI build system. > > If non-secure side RTOSes use fix values for tf-m dependencies then either build or regression tests may fail when tf-m updates dependencies. In order avoid this issue, I’m proposing the following > > > * Use a reStructuredText grid table to define dependencies > * Python script to produce a json file in the root directory when invoked as standalone script or to return a json string when imported into another python script > > I’ve created a patch set (https://review.trustedfirmware.org/c/trusted-firmware-m/+/2333) with above changes. I’d appreciate your inputs/feedback on this proposal. > > This will also ensure that dependencies are captured in one place (docs/user_guides/tfm_build_instruction.rst). > > Additionally, have another python script as “post-checkout” git hook which can parse dependencies in JSON format and clone them automatically when tf-m repository is cloned or when switching from master to feature branch and vice versa. > > Thanks, > Dev > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > -- > TF-M mailing list > TF-M@lists.trustedfirmware.org > https://lists.trustedfirmware.org/mailman/listinfo/tf-m

-- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m

Hi Devaraj,

Apologies for the late response, but I have been away in a workshop.

Based on the current feedback I believe that making cmake output the results is the way to go. It is simpler to implement and what most parties consider as reasonable implementation.

Adding a cmake target can be pretty well contained in a configuration file, and the responsibility to update the documentation should lie on the developers.

Regards, Minos ________________________________ From: Devaraj Ranganna Devaraj.Ranganna@arm.com Sent: 06 November 2019 11:18 To: Gyorgy Szing Gyorgy.Szing@arm.com; Minos Galanakis Minos.Galanakis@arm.com; tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com; Jaeden Amero Jaeden.Amero@arm.com Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

Any feedback on below proposal?

On 31/10/2019, 11:36, "Devaraj Ranganna" Devaraj.Ranganna@arm.com wrote:

Hi,

Any thoughts/feedback on separating build and post-build dependencies?

Thanks, Dev

On 23/10/2019, 16:04, "Devaraj Ranganna" Devaraj.Ranganna@arm.com wrote:

Hi,

Thanks for your inputs.

The original intention of the script was to not to modify the origin of the information i.e rst file as Minos quoted and also to avoid duplication of information. Considering all the new information, I think we should try to separate build and post-build dependencies (ex: srec_cat). This separation will allow us to handle all the build dependencies using "cmake" and then we can explore the possible option on how to handle post-build dependencies.

@Minos, I believe all sw dependencies defined in https://ci.trustedfirmware.org/job/tf-m-build-test-nightly/lastSuccessfulBui... are already validated in cmake. Adding other library dependencies into it will make it complete. This way we don’t need mention library dependency in document anymore and still achieve single source of information.

@George, I think first 3 question can be answered if we decide to go ahead with handling build dependencies with cmake.

Thanks, Dev

On 21/10/2019, 15:56, "TF-M on behalf of Gyorgy Szing via TF-M" <tf-m-bounces@lists.trustedfirmware.org on behalf of tf-m@lists.trustedfirmware.org> wrote:

Hi,

We need to keep an eye on some factors which the current prototype ignores. Some that come to my mind:

1. which component shall be the owner of sw dependency info? (Documentation, cmake, something else?). 2. how does the solution scale? (i.e. be able to handle platform specific dependencies). 3. how we handle build configuration specific dependencies? (i.e. if I don't build a service then some dependency is not needed). 4. As Minos mentioned some environment verification is already done by CMake. Is it worth to extract all dependency info (tooling + SW) into a dedicated place?

/George

-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Minos Galanakis via TF-M Sent: 21 October 2019 16:47 To: tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

Hi,

Making CMAKE able to output the dependencies it is expecting is the quickest path, but it is only able to resolve a subset of the requirements captured by the documenation. .https://ci.trustedfirmware.org/job/tf-m-build-test-nightly/lastSuccessfulBuild/artifact/build-docs/tf-m_documents/install/doc/user_guide/html/docs/user_guides/tfm_sw_requirement.html For example the cmake version itself, or tools like srec_cat used to produce MUSCA binaries. There are certain dependencies that are provided by other means ( package managers ) .

So while I agree that it looks quite odd, it is hierarchically the origin of the information we are trying to capture.

Minos ________________________________ From: TF-M tf-m-bounces@lists.trustedfirmware.org on behalf of Kumar Gala via TF-M tf-m@lists.trustedfirmware.org Sent: 21 October 2019 15:25 To: Devaraj Ranganna Devaraj.Ranganna@arm.com Cc: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

This feels backwards, parsing rst to determine build steps, dependancies doesn’t feel correct. I think it would be better to codify the build steps in scripts of CMake, etc than reference those in the .rst instead.

- k

> On Oct 21, 2019, at 9:03 AM, Devaraj Ranganna via TF-M tf-m@lists.trustedfirmware.org wrote: > > Hi, > > Currently tf-m dependencies are only listed in documentation file (docs/user_guides/tfm_build_instruction.rst) and there is no option to programmatically retrieve it. In order to integrate tf-m into non-secure side RTOSes, it is very essential to be able build tf-m automatically without manual intervention (may be as part of CI). > > The tf-m CI build system uses fixed values for dependencies and it is the responsibility of developers who update the dependencies to update the documentation and the CI build system. > > If non-secure side RTOSes use fix values for tf-m dependencies then either build or regression tests may fail when tf-m updates dependencies. In order avoid this issue, I’m proposing the following > > > * Use a reStructuredText grid table to define dependencies > * Python script to produce a json file in the root directory when invoked as standalone script or to return a json string when imported into another python script > > I’ve created a patch set (https://review.trustedfirmware.org/c/trusted-firmware-m/+/2333) with above changes. I’d appreciate your inputs/feedback on this proposal. > > This will also ensure that dependencies are captured in one place (docs/user_guides/tfm_build_instruction.rst). > > Additionally, have another python script as “post-checkout” git hook which can parse dependencies in JSON format and clone them automatically when tf-m repository is cloned or when switching from master to feature branch and vice versa. > > Thanks, > Dev > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > -- > TF-M mailing list > TF-M@lists.trustedfirmware.org > https://lists.trustedfirmware.org/mailman/listinfo/tf-m

-- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m

Devaraj,

Thank you for your contribution. I have posted some comments on the patch.

As per requested feedback I would ask to clarify what the task is:

1. The task is to simply rewind the CMAKE system to know where dependencies are expected to come from, and extract them if asked 2. The task is to make the CMAKE system able to also automatically download said dependencies 3. The task is to make the CMAKE system evaluate an existing enviroment and create json file with what is installed and what is missing.

To my understanding you need item #1 and we should keep it simple unless there is a real need for items 2, and 3. To that end my take is that the patch need to fulfill two criteria:

* Create a new cmake file where we are setting all dependencies, and convert existing files to refer to this file for version comparison on dependenciesversion from it * After this file has been converted to the core root of version resolution, we add an export to json method which will just format the data accordingly.

What should also be decided is if the new system will only support latest release or weather those dependency tags are fixed to the TF-M version.

That is my personal take, though, it would be good to have input from people with more exposure to the CMAKE/build system.

Regards Minos

________________________________ From: Devaraj Ranganna Devaraj.Ranganna@arm.com Sent: 19 November 2019 16:01 To: Minos Galanakis Minos.Galanakis@arm.com; Gyorgy Szing Gyorgy.Szing@arm.com; tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com; Jaeden Amero Jaeden.Amero@arm.com Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

Hi,

Apologies for late reply.

I’ve created a new patchset https://review.trustedfirmware.org/c/trusted-firmware-m/+/2562 considering all the feedback. I’ve Added two new cmake config options -DGEN_DEPS and -DDEPS_FILE, when specified cmake will export TF-M dependencies to the JSON file specified in option -DDEPS_FILE.

I’d appreciate any feedback on the new patchset.

Questions from George:

1. which component shall be the owner of sw dependency info? (Documentation, cmake, something else?). * cmake 2. how does the solution scale? (i.e. be able to handle platform specific dependencies). * Not possible with current patchset. Any suggestions? 3. how we handle build configuration specific dependencies? (i.e. if I don't build a service then some dependency is not needed). * Not possible with current patchset. . Any suggestions? 4. As Minos mentioned some environment verification is already done by CMake. Is it worth to extract all dependency info (tooling + SW) into a dedicated place? * With this patchset, we still need to maintain dependency info in documentation as the patchset only exports the dependencies and doesn’t manage them automatically.

Thanks,

Dev

From: Minos Galanakis Minos.Galanakis@arm.com Date: Wednesday, 6 November 2019 at 11:55 To: Devaraj Ranganna Devaraj.Ranganna@arm.com, Gyorgy Szing Gyorgy.Szing@arm.com, "tf-m@lists.trustedfirmware.org" tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com, Jaeden Amero Jaeden.Amero@arm.com Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

Hi Devaraj,

Apologies for the late response, but I have been away in a workshop.

Based on the current feedback I believe that making cmake output the results is the way to go. It is simpler to implement and what most parties consider as reasonable implementation.

Adding a cmake target can be pretty well contained in a configuration file, and the responsibility to update the documentation should lie on the developers.

Regards,

Minos

________________________________

From: Devaraj Ranganna Devaraj.Ranganna@arm.com Sent: 06 November 2019 11:18 To: Gyorgy Szing Gyorgy.Szing@arm.com; Minos Galanakis Minos.Galanakis@arm.com; tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com; Jaeden Amero Jaeden.Amero@arm.com Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

Any feedback on below proposal?

On 31/10/2019, 11:36, "Devaraj Ranganna" Devaraj.Ranganna@arm.com wrote:

Hi,

Any thoughts/feedback on separating build and post-build dependencies?

Thanks, Dev

On 23/10/2019, 16:04, "Devaraj Ranganna" Devaraj.Ranganna@arm.com wrote:

Hi,

Thanks for your inputs.

The original intention of the script was to not to modify the origin of the information i.e rst file as Minos quoted and also to avoid duplication of information. Considering all the new information, I think we should try to separate build and post-build dependencies (ex: srec_cat). This separation will allow us to handle all the build dependencies using "cmake" and then we can explore the possible option on how to handle post-build dependencies.

@Minos, I believe all sw dependencies defined in https://ci.trustedfirmware.org/job/tf-m-build-test-nightly/lastSuccessfulBui... are already validated in cmake. Adding other library dependencies into it will make it complete. This way we don’t need mention library dependency in document anymore and still achieve single source of information.

@George, I think first 3 question can be answered if we decide to go ahead with handling build dependencies with cmake.

Thanks, Dev

On 21/10/2019, 15:56, "TF-M on behalf of Gyorgy Szing via TF-M" <tf-m-bounces@lists.trustedfirmware.org on behalf of tf-m@lists.trustedfirmware.org> wrote:

Hi,

We need to keep an eye on some factors which the current prototype ignores. Some that come to my mind:

1. which component shall be the owner of sw dependency info? (Documentation, cmake, something else?). 2. how does the solution scale? (i.e. be able to handle platform specific dependencies). 3. how we handle build configuration specific dependencies? (i.e. if I don't build a service then some dependency is not needed). 4. As Minos mentioned some environment verification is already done by CMake. Is it worth to extract all dependency info (tooling + SW) into a dedicated place?

/George

-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Minos Galanakis via TF-M Sent: 21 October 2019 16:47 To: tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

Hi,

Making CMAKE able to output the dependencies it is expecting is the quickest path, but it is only able to resolve a subset of the requirements captured by the documenation. .https://ci.trustedfirmware.org/job/tf-m-build-test-nightly/lastSuccessfulBuild/artifact/build-docs/tf-m_documents/install/doc/user_guide/html/docs/user_guides/tfm_sw_requirement.html For example the cmake version itself, or tools like srec_cat used to produce MUSCA binaries. There are certain dependencies that are provided by other means ( package managers ) .

So while I agree that it looks quite odd, it is hierarchically the origin of the information we are trying to capture.

Minos ________________________________ From: TF-M tf-m-bounces@lists.trustedfirmware.org on behalf of Kumar Gala via TF-M tf-m@lists.trustedfirmware.org Sent: 21 October 2019 15:25 To: Devaraj Ranganna Devaraj.Ranganna@arm.com Cc: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Subject: Re: [TF-M] Python script to generate tf-m dependencies in JSON format

This feels backwards, parsing rst to determine build steps, dependancies doesn’t feel correct. I think it would be better to codify the build steps in scripts of CMake, etc than reference those in the .rst instead.

- k

> On Oct 21, 2019, at 9:03 AM, Devaraj Ranganna via TF-M tf-m@lists.trustedfirmware.org wrote: > > Hi, > > Currently tf-m dependencies are only listed in documentation file (docs/user_guides/tfm_build_instruction.rst) and there is no option to programmatically retrieve it. In order to integrate tf-m into non-secure side RTOSes, it is very essential to be able build tf-m automatically without manual intervention (may be as part of CI). > > The tf-m CI build system uses fixed values for dependencies and it is the responsibility of developers who update the dependencies to update the documentation and the CI build system. > > If non-secure side RTOSes use fix values for tf-m dependencies then either build or regression tests may fail when tf-m updates dependencies. In order avoid this issue, I’m proposing the following > > > * Use a reStructuredText grid table to define dependencies > * Python script to produce a json file in the root directory when invoked as standalone script or to return a json string when imported into another python script > > I’ve created a patch set (https://review.trustedfirmware.org/c/trusted-firmware-m/+/2333) with above changes. I’d appreciate your inputs/feedback on this proposal. > > This will also ensure that dependencies are captured in one place (docs/user_guides/tfm_build_instruction.rst). > > Additionally, have another python script as “post-checkout” git hook which can parse dependencies in JSON format and clone them automatically when tf-m repository is cloned or when switching from master to feature branch and vice versa. > > Thanks, > Dev > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > -- > TF-M mailing list > TF-M@lists.trustedfirmware.org > https://lists.trustedfirmware.org/mailman/listinfo/tf-m

-- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m