Hi,
I'm planning to settle down the first three patches for tools change of this topic, before Christmas. Because personally think they are very mature and close to merge. (And I've still working on the reset patches). So if there are no more new review comments, I'll go with the current reviews and try to merge it.
Best Regards, Kevin
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Kevin Peng (Arm Technology China) via TF-M Sent: Friday, December 6, 2019 2:41 PM To: DeMars, Alan ademars@ti.com; 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] [EXTERNAL] Re: out of tree build
I've created some patches for supporting this out of tree build:
https://review.trustedfirmware.org/q/topic:%22out_of_tree_build%22+(status:o...)
I firstly modified the tools/tfm_parse_manifest_list.py so that it can take customized secure partition manifest and list of files to generate and output the files to a specified directory. You can even "append" your manifests or files to generate to the default lists under tools. The format of the yamls are the same.
The tfm_parse_manifest_list.py script will also create a CMakeLists.inc which contains all the paths of generated header. And the paths are added to include search path. Other CMake projects can include this file to find generated headers.
And then the generated files are moved to a dedicated folder to solve the issue mentioned below (generated headers cannot be included).
Finally, the build system is able to call the parse tool to generate files in build time(can be disabled) and use the new generated files to build.
You can set the optional arguments for the parse tool by setting CMake arguments:
cmake -DTFM_MANIFEST(_APPEND)=some_manifest -DTFM_GEN_LIST(_APPEND)=some_generated_file -DTFM_GEN_DIR=... -G'Unix Makefiles' -DTARGET_PLATFORM=AN521 -D COMPILER=ARMCLANG -DPROJ_CONFIG=`readlink -f ConfigXXX.cmake` TFM_ROOT
cmake –build
Or you can pre-generate the files, disable build time generation and tell cmake the output directory only.
Comments and discussions are welcome.
Best Regards,
Kevin
-----Original Message-----
From: DeMars, Alan ademars@ti.com
Sent: Saturday, November 23, 2019 12:33 AM
To: Kevin Peng (Arm Technology China) Kevin.Peng@arm.com
Cc: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org
Subject: RE: [TF-M] [EXTERNAL] Re: out of tree build
In our use case, we added complete paths to the build area's corresponding directory to the embedded include paths and removed the default generated content from our git repo. Consequently only the newly generated content is found, wherever it happens to be placed.
Alan
-----Original Message-----
From: TF-M [mailto:tf-m-bounces@lists.trustedfirmware.org] On Behalf Of Kevin Peng (Arm Technology China) via TF-M
Sent: Friday, November 22, 2019 2:15 AM
To: tf-m@lists.trustedfirmware.org
Cc: nd
Subject: Re: [TF-M] [EXTERNAL] Re: out of tree build
Hi all,
I'm having issues on supporting the out of tree build.
So I'd like to propose some additional changes.
What does the out of tree build currently support is that:
The user uses the tfm_parse_manifest_list.py with the custom manifests to generate the customized files to a directory out of TF-M.
And the TF-M build system uses the customized files rather than the pre-generated ones in the TF-M for building.
The problem is that some of the source files include the generated headers using the relative path to where the source itself is.
And the directory of the source file is always searched first.
This makes it impossible to use the generated files outside TF-M even the custom output directory is added to searching list.
So I suggest to put the generated files to a dedicated directory in TF-M.
Then for the default build, add the directory and its subdirectories to the search list for headers.
For out of tree build, use the specified directory and its subdirectories instead.
Then there would only one header file with the same name in all the directories for searching headers.
Any concerns, thoughts or suggestions?
Best Regards,
Kevin
-----Original Message-----
From: Kevin Peng (Arm Technology China)
Sent: Friday, November 22, 2019 3:45 PM
To: DeMars, Alan ademars@ti.com; tf-m@lists.trustedfirmware.org
Cc: nd nd@arm.com
Subject: RE: [EXTERNAL] Re: [TF-M] out of tree build
Hi Alan,
I've created a patch for the tfm_parse_manifest_list.py as the first step for the out of tree build support:
https://review.trustedfirmware.org/c/trusted-firmware-m/+/2606
The changes for build system is ongoing.
The functionality changes for the tfm_parse_manifest_list.py is the same as your version.
Please have a review and help verify it if possible.
Comments from anyone else are also welcome.
Best Regards,
Kevin
-----Original Message-----
From: Kevin Peng (Arm Technology China)
Sent: Thursday, November 14, 2019 3:31 PM
To: DeMars, Alan ademars@ti.com
Cc: tf-m@lists.trustedfirmware.org; nd nd@arm.com
Subject: RE: [EXTERNAL] Re: [TF-M] out of tree build
OK.
It's much clearer for me now. Thanks.
Best Regards,
Kevin
-----Original Message-----
From: DeMars, Alan ademars@ti.com
Sent: Thursday, November 14, 2019 10:34 AM
To: Kevin Peng (Arm Technology China) Kevin.Peng@arm.com
Cc: tf-m@lists.trustedfirmware.org; nd nd@arm.com
Subject: Re: [EXTERNAL] Re: [TF-M] out of tree build
1) I verified the tfm_parse_manifest.py script changes using the master branch’s version of the tfm_manifest_list.yaml and tfm_generated_file_list.yaml files.
2) Yes, I’m building using the unmodified master branch’s cmake build system.
3) Just as all other build artifacts are placed in the user’s build directory and NOT in the source tree, the generated files should also. Otherwise, the user is required to update the source tree’s tfm_manifest_list.yaml and tfm_generated_file_list.yaml files to build a custom SPE, with the generated files also ending up in the source tree. The files necessary to create a custom SPE should not be kept in the source tree. Nor should the generated content be inserted in the source tree. This creates headaches for maintaining the original source tree.
Alan
On Nov 13, 2019, at 1:57 AM, Kevin Peng (Arm Technology China) via TF-M tf-m@lists.trustedfirmware.org wrote:
Hi Alan,
I checked your modified tfm_parse_manifest_list.py.
It basically uses the input tfm_manifest_list.yaml and tfm_generated_file_list.yaml to generate the TF-M auto generated files to the third input.
I'm sorry I was not in the workshop. So I have some questions.
- Do you have your own tfm_manifest_list.yaml and tfm_generated_file_list.yaml that is not upstreamed in TF-M?
- Are you using the TF-M provided CMake build system?
- Do you only need the files generated by tfm_parse_manifest_list.py to be out of the TF-M source directory?
Could you provide some details about your requirements. Thanks.
Best Regards,
Kevin
-----Original Message-----
From: Kevin Peng (Arm Technology China)
Sent: Wednesday, November 13, 2019 10:27 AM
To: tf-m@lists.trustedfirmware.org
Cc: nd nd@arm.com
Subject: RE: out of tree build
Hi Alan,
I'm following on this and will get you updated on any progress.
Best Regards,
Kevin
-----Original Message-----
From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of DeMars, Alan via TF-M
Sent: Thursday, November 7, 2019 12:42 PM
To: Abhishek Pandit Abhishek.Pandit@arm.com
Cc: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org
Subject: Re: [TF-M] out of tree build
Trying again after renaming tfm_parse_manifest_list.py to tfm_parse_manifest_list.txt so it wouldn't get deleted.
With the attached modified secure_fw/CMakeLists.txt and tools/tfm_parse_manifest_list.py files, I am able to redirect the generated files using the following command line:
python3 <full_path_to_your>/tfm_parse_manifest_list.py <full_path_to_your>/tfm_manifest_list.yaml <full_path_to_your>/tfm_generated_file_list.yaml <full_path_to_your>/<build_dir>
And then build the ConfigCoreIPC artifacts within <your_build_dir> using the following cmake line:
cmake -G"Unix Makefiles" -DPROJ_CONFIG=`readlink -f ../configs/ConfigCoreIPC.cmake` -DREMOTE_GEN_DIR=<full_path_to_your><build_dir> -DTARGET_PLATFORM=AN521 -DCOMPILER=GNUARM ../
The changes to tfm_parse_manifest_list.py are backward compatible with the standard usage.
Alan
-----Original Message-----
From: TF-M [mailto:tf-m-bounces@lists.trustedfirmware.org] On Behalf Of DeMars, Alan via TF-M
Sent: Monday, November 4, 2019 4:36 PM
To: Abhishek Pandit; tf-m@lists.trustedfirmware.org
Subject: [EXTERNAL] Re: [TF-M] out of tree build
Abishek,
Yes, we have modified tfm_gen.py and tfm_parse_manifest_list.py to support redirecting the destination of the generated template files into a command line provided destination build directory.
A corresponding change to our platform's platform/ext/xyz.cmake file was also required to add the path to the root of the build directory to the embedded_include_directories() list as well as the paths to the generated linker command files.
I was not planning to provide these changes as a patch for review as I am very unsure of the applicability of this to other platforms. Also, I was fairly certain that given my very poor understanding of the CMake build system, my approach to the problem was not utilizing features present in CMake that would make the job simpler and more extensible.
Since the topic came up at the conference and you already seemed willing to address the out-of-tree build problem that the templates lead to, I assumed you folks would find a simple and elegant solution that would save me the embarrassment of exposing my lack of CMake expertise.
Alan
-----Original Message-----
From: Abhishek Pandit [mailto:Abhishek.Pandit@arm.com]
Sent: Monday, November 4, 2019 3:08 PM
To: DeMars, Alan; tf-m@lists.trustedfirmware.org
Subject: [EXTERNAL] RE: out of tree build
Hi Alan,
Not sure if I remember the exact detail from the workshop last week, but did you mention that you have a prototype for this? If so, are you planning to push a patch for review?
Thanks,
Abhishek
-----Original Message-----
From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of DeMars, Alan via TF-M
Sent: 01 November 2019 16:29
To: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org
Subject: [TF-M] out of tree build
Please modify the template generators to support out of tree build, and modify the CMake files to add the necessary include paths so that the files that include the template-generated files can find them.
Alan
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Hi,
The changes of the manifest parsing tool have been merged. The following changes for the build system is updated a lot. So I'm opening for comments for another week. Any kinds of feedbacks are welcome.
https://review.trustedfirmware.org/q/topic:%22out_of_tree_build%22+(status:o...)
Best Regards, Kevin
-----Original Message----- From: Kevin Peng Sent: Thursday, December 19, 2019 6:53 AM To: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: RE: [TF-M] [EXTERNAL] Re: out of tree build
Hi,
I'm planning to settle down the first three patches for tools change of this topic, before Christmas. Because personally think they are very mature and close to merge. (And I've still working on the reset patches). So if there are no more new review comments, I'll go with the current reviews and try to merge it.
Best Regards, Kevin
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Kevin Peng (Arm Technology China) via TF-M Sent: Friday, December 6, 2019 2:41 PM To: DeMars, Alan ademars@ti.com; 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] [EXTERNAL] Re: out of tree build
I've created some patches for supporting this out of tree build:
https://review.trustedfirmware.org/q/topic:%22out_of_tree_build%22+(status:o...)
I firstly modified the tools/tfm_parse_manifest_list.py so that it can take customized secure partition manifest and list of files to generate and output the files to a specified directory. You can even "append" your manifests or files to generate to the default lists under tools. The format of the yamls are the same.
The tfm_parse_manifest_list.py script will also create a CMakeLists.inc which contains all the paths of generated header. And the paths are added to include search path. Other CMake projects can include this file to find generated headers.
And then the generated files are moved to a dedicated folder to solve the issue mentioned below (generated headers cannot be included).
Finally, the build system is able to call the parse tool to generate files in build time(can be disabled) and use the new generated files to build.
You can set the optional arguments for the parse tool by setting CMake arguments:
cmake -DTFM_MANIFEST(_APPEND)=some_manifest -DTFM_GEN_LIST(_APPEND)=some_generated_file -DTFM_GEN_DIR=... -G'Unix Makefiles' -DTARGET_PLATFORM=AN521 -D COMPILER=ARMCLANG -DPROJ_CONFIG=`readlink -f ConfigXXX.cmake` TFM_ROOT
cmake –build
Or you can pre-generate the files, disable build time generation and tell cmake the output directory only.
Comments and discussions are welcome.
Best Regards,
Kevin
-----Original Message-----
From: DeMars, Alan ademars@ti.com
Sent: Saturday, November 23, 2019 12:33 AM
To: Kevin Peng (Arm Technology China) Kevin.Peng@arm.com
Cc: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org
Subject: RE: [TF-M] [EXTERNAL] Re: out of tree build
In our use case, we added complete paths to the build area's corresponding directory to the embedded include paths and removed the default generated content from our git repo. Consequently only the newly generated content is found, wherever it happens to be placed.
Alan
-----Original Message-----
From: TF-M [mailto:tf-m-bounces@lists.trustedfirmware.org] On Behalf Of Kevin Peng (Arm Technology China) via TF-M
Sent: Friday, November 22, 2019 2:15 AM
To: tf-m@lists.trustedfirmware.org
Cc: nd
Subject: Re: [TF-M] [EXTERNAL] Re: out of tree build
Hi all,
I'm having issues on supporting the out of tree build.
So I'd like to propose some additional changes.
What does the out of tree build currently support is that:
The user uses the tfm_parse_manifest_list.py with the custom manifests to generate the customized files to a directory out of TF-M.
And the TF-M build system uses the customized files rather than the pre-generated ones in the TF-M for building.
The problem is that some of the source files include the generated headers using the relative path to where the source itself is.
And the directory of the source file is always searched first.
This makes it impossible to use the generated files outside TF-M even the custom output directory is added to searching list.
So I suggest to put the generated files to a dedicated directory in TF-M.
Then for the default build, add the directory and its subdirectories to the search list for headers.
For out of tree build, use the specified directory and its subdirectories instead.
Then there would only one header file with the same name in all the directories for searching headers.
Any concerns, thoughts or suggestions?
Best Regards,
Kevin
-----Original Message-----
From: Kevin Peng (Arm Technology China)
Sent: Friday, November 22, 2019 3:45 PM
To: DeMars, Alan ademars@ti.com; tf-m@lists.trustedfirmware.org
Cc: nd nd@arm.com
Subject: RE: [EXTERNAL] Re: [TF-M] out of tree build
Hi Alan,
I've created a patch for the tfm_parse_manifest_list.py as the first step for the out of tree build support:
https://review.trustedfirmware.org/c/trusted-firmware-m/+/2606
The changes for build system is ongoing.
The functionality changes for the tfm_parse_manifest_list.py is the same as your version.
Please have a review and help verify it if possible.
Comments from anyone else are also welcome.
Best Regards,
Kevin
-----Original Message-----
From: Kevin Peng (Arm Technology China)
Sent: Thursday, November 14, 2019 3:31 PM
To: DeMars, Alan ademars@ti.com
Cc: tf-m@lists.trustedfirmware.org; nd nd@arm.com
Subject: RE: [EXTERNAL] Re: [TF-M] out of tree build
OK.
It's much clearer for me now. Thanks.
Best Regards,
Kevin
-----Original Message-----
From: DeMars, Alan ademars@ti.com
Sent: Thursday, November 14, 2019 10:34 AM
To: Kevin Peng (Arm Technology China) Kevin.Peng@arm.com
Cc: tf-m@lists.trustedfirmware.org; nd nd@arm.com
Subject: Re: [EXTERNAL] Re: [TF-M] out of tree build
1) I verified the tfm_parse_manifest.py script changes using the master branch’s version of the tfm_manifest_list.yaml and tfm_generated_file_list.yaml files.
2) Yes, I’m building using the unmodified master branch’s cmake build system.
3) Just as all other build artifacts are placed in the user’s build directory and NOT in the source tree, the generated files should also. Otherwise, the user is required to update the source tree’s tfm_manifest_list.yaml and tfm_generated_file_list.yaml files to build a custom SPE, with the generated files also ending up in the source tree. The files necessary to create a custom SPE should not be kept in the source tree. Nor should the generated content be inserted in the source tree. This creates headaches for maintaining the original source tree.
Alan
On Nov 13, 2019, at 1:57 AM, Kevin Peng (Arm Technology China) via TF-M tf-m@lists.trustedfirmware.org wrote:
Hi Alan,
I checked your modified tfm_parse_manifest_list.py.
It basically uses the input tfm_manifest_list.yaml and tfm_generated_file_list.yaml to generate the TF-M auto generated files to the third input.
I'm sorry I was not in the workshop. So I have some questions.
- Do you have your own tfm_manifest_list.yaml and tfm_generated_file_list.yaml that is not upstreamed in TF-M?
- Are you using the TF-M provided CMake build system?
- Do you only need the files generated by tfm_parse_manifest_list.py to be out of the TF-M source directory?
Could you provide some details about your requirements. Thanks.
Best Regards,
Kevin
-----Original Message-----
From: Kevin Peng (Arm Technology China)
Sent: Wednesday, November 13, 2019 10:27 AM
To: tf-m@lists.trustedfirmware.org
Cc: nd nd@arm.com
Subject: RE: out of tree build
Hi Alan,
I'm following on this and will get you updated on any progress.
Best Regards,
Kevin
-----Original Message-----
From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of DeMars, Alan via TF-M
Sent: Thursday, November 7, 2019 12:42 PM
To: Abhishek Pandit Abhishek.Pandit@arm.com
Cc: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org
Subject: Re: [TF-M] out of tree build
Trying again after renaming tfm_parse_manifest_list.py to tfm_parse_manifest_list.txt so it wouldn't get deleted.
With the attached modified secure_fw/CMakeLists.txt and tools/tfm_parse_manifest_list.py files, I am able to redirect the generated files using the following command line:
python3 <full_path_to_your>/tfm_parse_manifest_list.py <full_path_to_your>/tfm_manifest_list.yaml <full_path_to_your>/tfm_generated_file_list.yaml <full_path_to_your>/<build_dir>
And then build the ConfigCoreIPC artifacts within <your_build_dir> using the following cmake line:
cmake -G"Unix Makefiles" -DPROJ_CONFIG=`readlink -f ../configs/ConfigCoreIPC.cmake` -DREMOTE_GEN_DIR=<full_path_to_your><build_dir> -DTARGET_PLATFORM=AN521 -DCOMPILER=GNUARM ../
The changes to tfm_parse_manifest_list.py are backward compatible with the standard usage.
Alan
-----Original Message-----
From: TF-M [mailto:tf-m-bounces@lists.trustedfirmware.org] On Behalf Of DeMars, Alan via TF-M
Sent: Monday, November 4, 2019 4:36 PM
To: Abhishek Pandit; tf-m@lists.trustedfirmware.org
Subject: [EXTERNAL] Re: [TF-M] out of tree build
Abishek,
Yes, we have modified tfm_gen.py and tfm_parse_manifest_list.py to support redirecting the destination of the generated template files into a command line provided destination build directory.
A corresponding change to our platform's platform/ext/xyz.cmake file was also required to add the path to the root of the build directory to the embedded_include_directories() list as well as the paths to the generated linker command files.
I was not planning to provide these changes as a patch for review as I am very unsure of the applicability of this to other platforms. Also, I was fairly certain that given my very poor understanding of the CMake build system, my approach to the problem was not utilizing features present in CMake that would make the job simpler and more extensible.
Since the topic came up at the conference and you already seemed willing to address the out-of-tree build problem that the templates lead to, I assumed you folks would find a simple and elegant solution that would save me the embarrassment of exposing my lack of CMake expertise.
Alan
-----Original Message-----
From: Abhishek Pandit [mailto:Abhishek.Pandit@arm.com]
Sent: Monday, November 4, 2019 3:08 PM
To: DeMars, Alan; tf-m@lists.trustedfirmware.org
Subject: [EXTERNAL] RE: out of tree build
Hi Alan,
Not sure if I remember the exact detail from the workshop last week, but did you mention that you have a prototype for this? If so, are you planning to push a patch for review?
Thanks,
Abhishek
-----Original Message-----
From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of DeMars, Alan via TF-M
Sent: 01 November 2019 16:29
To: 'tf-m@lists.trustedfirmware.org' tf-m@lists.trustedfirmware.org
Subject: [TF-M] out of tree build
Please modify the template generators to support out of tree build, and modify the CMake files to add the necessary include paths so that the files that include the template-generated files can find them.
Alan
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
tf-m@lists.trustedfirmware.org
-
Kevin Peng