Hi Minos,
Thanks for the detailed reply/explanation. This sounds similar to the CI setup for Zephyr where I test changes locally first.
I'm happy to put together a change request for this, based on any feedback here (if it's deemed worth merging in, etc.): https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
Best regards, Kevin
On Mon, 5 Aug 2019 at 15:34, Minos Galanakis via TF-M tf-m@lists.trustedfirmware.org wrote:
The CI comprises of three modules. The Jenkins logic, the Python Scripts and the docker build slave.
The environment is provided by the docker build slave and provisioned into the pipeline flow when the venv is created for each stage:
virtualenv -p python3 ${VENV_P3_NAME} --system-site-packages
No requirements are installed at the Jenkins Stage, but if needed as an one-off (i.e. for staging purposes), the design supports it.
The case for installing Python requirements on the fly using requirements.txt
Extending requirements dynamically on the fly, can be quite a challenge due to the way Jenkins handles the absolute resolution of workspace directory on each step. In short Python’s virtual-environment stores the configuration paths in absolute format, while Jenkins is not guaranteed to give you the same reference to a working directory in consecutive calls in the pipeline.
So if you create a venv at stage 1, which evaluates ~/ as /server/workspace/fubar-job/venv/.. and then attempt to call it in a following parallel step, the code may be located at /server/workspace/1/fubar-job/venv
At this point you can either create the VENV in each stage, and reinstall the requirements, effectively wasting bandwidth or hack it by piping everything in SED before activating to ensure the path is resolved correctly.
For that purposes the ci-scripts level requirements.txt will be deprecated in the next feature update.
How should a user access or modify the TF-M Build environment.
Environment will be established at the docker build stage.
https://review.trustedfirmware.org/admin/repos/ci/dockerfiles
And more specifically the requirements for Python3 in this file:
https://git.trustedfirmware.org/ci/dockerfiles.git/tree/xenial-amd64-tf-m-bu...
After it has been updated and merged you should raise a ticket and request the image to get rebuilt, or if it is not very critical, wait for some other party change to trigger the build.
The process requires creating local docker image, meant to test your changes but also allowing you to access the TF-M build environment as deployed on the CI. You can do that following the steps below:
# Get the docker image
$ git clone https://review.trustedfirmware.org/ci/dockerfiles && cd dockerfiles/xenial-amd64-tf-m-build
# Edit the entry point to convert it not to be a jenkins-slave
$ vi Dockerfile
# change ENTRYPOINT ["/usr/local/bin/jenkins-slave"] to ENTRYPOINT [/bin/bash"], save exit
# Build the image
$ docker build ./
# Find the image hash id
$ docker image ls
# Run an interactive bash shell, mounting a local directory as /opt/openci in instance (if required to share files)
$ docker run -it --name tf-m-build-env -v /YOUR_CUSTOM_PATH:/opt/openci 10bcb173cd39
# You can relaunch that instance in the future by starting it again.
$ docker start && docker -exec -it tf-m-build-env /bin/bash
Please let me know if you need more clarity or guidance on how to handle modifications on the CI.
Regards,
Minos Galanakis
From: TF-M tf-m-bounces@lists.trustedfirmware.org on behalf of Antonio De Angelis via TF-M tf-m@lists.trustedfirmware.org Sent: 02 August 2019 14:57 To: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] Changes to CI for python dependencies
+Minos now
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Antonio De Angelis via TF-M Sent: 02 August 2019 14:47 To: tf-m@lists.trustedfirmware.org Subject: Re: [TF-M] Changes to CI for python dependencies
Minos, could you have a look at this?
Thanks, Antonio
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Kevin Townsend via TF-M Sent: 02 August 2019 12:44 To: Thomas Törnblom via TF-M tf-m@lists.trustedfirmware.org Subject: [TF-M] Changes to CI for python dependencies
In an effort to migrate to the more modern 'cryptography' module in imgtool.py (which mcuboot has already switched to upstream), I created a change request here: https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
The change fails in CI, however, due to the missing cryptography module in the CI build environment: https://ci.trustedfirmware.org/job/tf-m-build-test-review/1740/artifact/buil...
This brings up the following issues:
- How can/should changes be made to the CI build environment?
- Can the overall TF-M installation process be improved automating Python module installation via a requirements.txt file?
Adding a requirements.txt file means that file could be run when the CI environment starts a new test build, taking into account any dependency changes that are part of the change request (version updates, etc.).
This would also have the positive side effect of users no longer having to scan through tfm_sw_requirement.rst to see what they don't have installed, or parse build failures for missing module names.
I'm happy to make a new change request adding a requirements.txt file, and update the documentation accordingly, but t's not clear to me how to propose the required changes to the CI setup?
Best regards, Kevin Townsend -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Kevin,
I think it is a good idea to submit the docker patch.
When a dependency has a reasonable footprint and more than one teams can benefit by using it, it should/could be provided by the build enviroment. Ideally cryptography and pycrypto modules can both be included without conflicts in order to provide a reasonable migration window.
Best regards Minos Galanakis ________________________________ From: Kevin Townsend kevin.townsend@linaro.org Sent: 05 August 2019 15:47 To: Minos Galanakis Minos.Galanakis@arm.com Cc: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org; Antonio De Angelis Antonio.DeAngelis@arm.com; nd nd@arm.com Subject: Re: [TF-M] Changes to CI for python dependencies
Hi Minos,
Thanks for the detailed reply/explanation. This sounds similar to the CI setup for Zephyr where I test changes locally first.
I'm happy to put together a change request for this, based on any feedback here (if it's deemed worth merging in, etc.): https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
Best regards, Kevin
On Mon, 5 Aug 2019 at 15:34, Minos Galanakis via TF-M tf-m@lists.trustedfirmware.org wrote:
The CI comprises of three modules. The Jenkins logic, the Python Scripts and the docker build slave.
The environment is provided by the docker build slave and provisioned into the pipeline flow when the venv is created for each stage:
virtualenv -p python3 ${VENV_P3_NAME} --system-site-packages
No requirements are installed at the Jenkins Stage, but if needed as an one-off (i.e. for staging purposes), the design supports it.
The case for installing Python requirements on the fly using requirements.txt
Extending requirements dynamically on the fly, can be quite a challenge due to the way Jenkins handles the absolute resolution of workspace directory on each step. In short Python’s virtual-environment stores the configuration paths in absolute format, while Jenkins is not guaranteed to give you the same reference to a working directory in consecutive calls in the pipeline.
So if you create a venv at stage 1, which evaluates ~/ as /server/workspace/fubar-job/venv/.. and then attempt to call it in a following parallel step, the code may be located at /server/workspace/1/fubar-job/venv
At this point you can either create the VENV in each stage, and reinstall the requirements, effectively wasting bandwidth or hack it by piping everything in SED before activating to ensure the path is resolved correctly.
For that purposes the ci-scripts level requirements.txt will be deprecated in the next feature update.
How should a user access or modify the TF-M Build environment.
Environment will be established at the docker build stage.
https://review.trustedfirmware.org/admin/repos/ci/dockerfiles
And more specifically the requirements for Python3 in this file:
https://git.trustedfirmware.org/ci/dockerfiles.git/tree/xenial-amd64-tf-m-bu...
After it has been updated and merged you should raise a ticket and request the image to get rebuilt, or if it is not very critical, wait for some other party change to trigger the build.
The process requires creating local docker image, meant to test your changes but also allowing you to access the TF-M build environment as deployed on the CI. You can do that following the steps below:
# Get the docker image
$ git clone https://review.trustedfirmware.org/ci/dockerfiles && cd dockerfiles/xenial-amd64-tf-m-build
# Edit the entry point to convert it not to be a jenkins-slave
$ vi Dockerfile
# change ENTRYPOINT ["/usr/local/bin/jenkins-slave"] to ENTRYPOINT [/bin/bash"], save exit
# Build the image
$ docker build ./
# Find the image hash id
$ docker image ls
# Run an interactive bash shell, mounting a local directory as /opt/openci in instance (if required to share files)
$ docker run -it --name tf-m-build-env -v /YOUR_CUSTOM_PATH:/opt/openci 10bcb173cd39
# You can relaunch that instance in the future by starting it again.
$ docker start && docker -exec -it tf-m-build-env /bin/bash
Please let me know if you need more clarity or guidance on how to handle modifications on the CI.
Regards,
Minos Galanakis
From: TF-M tf-m-bounces@lists.trustedfirmware.org on behalf of Antonio De Angelis via TF-M tf-m@lists.trustedfirmware.org Sent: 02 August 2019 14:57 To: tf-m@lists.trustedfirmware.org tf-m@lists.trustedfirmware.org Cc: nd nd@arm.com Subject: Re: [TF-M] Changes to CI for python dependencies
+Minos now
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Antonio De Angelis via TF-M Sent: 02 August 2019 14:47 To: tf-m@lists.trustedfirmware.org Subject: Re: [TF-M] Changes to CI for python dependencies
Minos, could you have a look at this?
Thanks, Antonio
-----Original Message----- From: TF-M tf-m-bounces@lists.trustedfirmware.org On Behalf Of Kevin Townsend via TF-M Sent: 02 August 2019 12:44 To: Thomas Törnblom via TF-M tf-m@lists.trustedfirmware.org Subject: [TF-M] Changes to CI for python dependencies
In an effort to migrate to the more modern 'cryptography' module in imgtool.py (which mcuboot has already switched to upstream), I created a change request here: https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
The change fails in CI, however, due to the missing cryptography module in the CI build environment: https://ci.trustedfirmware.org/job/tf-m-build-test-review/1740/artifact/buil...
This brings up the following issues:
- How can/should changes be made to the CI build environment?
- Can the overall TF-M installation process be improved automating Python module installation via a requirements.txt file?
Adding a requirements.txt file means that file could be run when the CI environment starts a new test build, taking into account any dependency changes that are part of the change request (version updates, etc.).
This would also have the positive side effect of users no longer having to scan through tfm_sw_requirement.rst to see what they don't have installed, or parse build failures for missing module names.
I'm happy to make a new change request adding a requirements.txt file, and update the documentation accordingly, but t's not clear to me how to propose the required changes to the CI setup?
Best regards, Kevin Townsend -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m -- TF-M mailing list TF-M@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-m
tf-m@lists.trustedfirmware.org
-
Kevin Townsend -
Minos Galanakis