- Tf-openci - lists.trustedfirmware.org

unannounced downtime on review.trustedfirmware.org

by Kelley Spoon

Greetings, Yesterday the gerrit server on review.trustedfirmware.org experienced significant downtime outside of any announced or scheduled maintenance windows starting around 5pm UTC. The downtime was caused by ansible deployment that was run to make some configuration changes and install a plugin. The deployment made several unintentional upgrades to other external plugins which were incompatible with our environment and prevented gerrit from restarting. As a result, for a 12 hour period the gerrit server was either off line or highly unstable. As of 5am UTC today, gerrit had been returned to its original state and resumed stable operation. I apologize for the unannounced downtime and any inconvenience that it caused, and we are reviewing the events to ensure a similar situation doesn't occur again, Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 4 months

1
0
0 0

OpenCI FVP Tuxsuite AWS Performance testing

by Joanna Farley

FYI at the weekend I ran an OpenCI Performance test to see if the Tuxsuite AWS changes for running FVP based TF-A tests helps with the management of load and execution times over running the FVP tests on physical servers. In short it does help! See https://discord.com/channels/1106321706588577904/1210643193834508338 for details and analysis. Joanna IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 year, 4 months

1
0
0 0

[Maintenance] - tf.validation.linaro.org down on 2023-02-09 9:30AM

by Luís Tonicha

Hello all, My name is Luís and I'm part of the Deployments team. We will be migrating the TF LAVA instance from its current deployment to our new, Kubernetes-based platform, this Friday (09/02) at 9:30 (AM) UTC. This means that the tf.validation.linaro.org service will be down for the expected downtime of 2 to 3 hours. This is just a migration so no down/upgrades or configuration changes to LAVA will happen. Once the service is back online we'll announce it. Thanks for your collaboration. Regards, -- Luís Tonicha <luis.tonicha(a)linaro.org>

1 year, 5 months

1
2
0 0

[Maintenance] - urgent downtime for all Jenkins servers today 2024-01-29

by Kelley Spoon

Greetings All, The Jenkins servers on ci-staging.trustedfirmware ci.trustedfirmware.org and mbedtls.trustedfirmware.org will all be put into safe mode in a few moments in preparation for a shutdown around 6pm UTC today. The purpose of this is to perform an upgrade to the server to address a critical CVE discovered for the version of Jenkins running on all servers. Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 5 months

1
1
0 0

TF-A CI TuxSuite integration - official launch

by Paul Sokolovsky

Dear Colleagues, Thanks for your patience while the latest TuxSuite integration issues were worked our during recent weeks. Last of the systematic FVP integration issues were addressed end of last week, and since the last weekend, the system was in the "soft launch" mode to see if there're any remaining issues. Some intermittent issues were addressed during this week. As of now, all the main TF-A CI jobs should be converted, specifically: * Mainline daily builds * Mainline L1/L2 patch builds * LTS2.8 daily builds * LTS2.8 L1/L2 patch builds Smaller related issues should also be addressed by now (in particular https://linaro.atlassian.net/browse/TFC-573 "Make sure that Tux test failure leads to Jenkins build being set to Failed status" which a subject of explicit request). But thanks for understanding that we may still need some tweaks here and there during the coming week(s). We're launching integration with the highly scalable TuxSuite backend well in advance of the next mainline release, and just in time for the anticipated second LTS branch being forked. All that hopefully gives us enough time to test the new system further and make any following adjustments as needed. But my testing shows that the main "selling points" of the new system are already at play - higher scalability and improved stability due to avoidance of common testing bottlenecks we experienced during recent times as TF CI coverage and load grew in high tempo. Please feel free to share your feedback and/or questions. Thanks, Linaro TF CI maintenance team Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

1 year, 5 months

1
0
0 0

TF MISRA tool, ECLAIR, upgraded to 3.13

by Paul Sokolovsky

Hello, ECLAIR version 3.13 was released end of last year, and as of now, both TF-A and TF-M CI jobs are upgraded to this version. The jobs and reports seem to be well, but please let me know if you see any issues. Just in case, we keep docker images with the previous version around, so if there are issues or a need for detailed comparison of result of old vs new version, we can easily do that. Thanks, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

1 year, 5 months

1
0
0 0

[Maintenance] - ci.trustedfirmware.org and mbedtls.trustedfirmware.org down time on 2024-01-13

by Kelley Spoon

Hello All, The servers will stop processing jobs on 2024-1-20 at around 2:00pm UTC, as the servers will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins and some of its plugins to the latest LTS release. Start: 2024-1-13 03:00pm UTC End: 2024-1-13 07:00pm UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 6 months

1
1
0 0

Deploying OpenCI TuxSuite FVP integration

by Paul Sokolovsky

Hello, Overloading and scalability issues have been a known issue with OpenCI for a long time, usually happening around the release cycles due to elevated CI activity, but also occurring from time to time during normal workflow too. That's why last quarter we worked on an actionable plan to leverage TuxSuite, a recent Linaro technology for cloud-based building and testing, which proved itself well with other projects. For this initial pilot project, we're looking to route TF-A FVP tests (90+% of our test load) to TuxSuite away from LAVA, to alleviate load on existing build and physical device test infrastructure. It was actively worked on during this month, and I'm happy to to report that initial development and preliminary testing on staging show encouraging results. I'd like to perform larger-scale testing on staging yet, but otherwise think that we should be ready to deploy to production next. As it's a holiday season with not many working days left, I'm sending this a bit earlier to make sure it's not forgotten or comes as a surprise later. The plan is otherwise to test on staging today and/or over weekend, and then proceed with deployment and validating on production next week(s). Please let me know if you have questions or concerns. Happy holidays, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

1 year, 6 months

2
3
0 0

[Maintenance] - ci.staging.trustedfirmware.org down time 2023-12-20

by Kelley Spoon

Hello All, The server will stop processing jobs on 2023-12-20 at around 6:00pm UTC, as the server will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins and some of its plugins to the latest LTS release. Start: 2023-12-20 07:00pm UTC End: 2023-12-20 09:00 UTC If this upgrade remains stable in staging, we will plan to upgrade ci.trustedfirmware.org and mbedtls.trustedfirmware.org shortly after 2024-01-03. Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 6 months

1
1
0 0

[Maintenance] - ci.trustedfirmware.org down time 2023-10-13

by Kelley Spoon

Hello All, The jenkins server will stop processing jobs on 2023-10-13 at around 21:00 UTC, as the server will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins to address some minor security vulnerabilities and upgrade internal dependencies off of obsolete versions. This upgrade will be to the same version that's been running on staging for the past week and a half. Start: 2023-10-13 21:00 UTC Finish: 2023-10-13 23:00 UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 9 months

1
1
0 0

2025

2024

2023

2022

Tf-openci