- Tf-openci - lists.trustedfirmware.org

[Maintenance] - tf.validation.linaro.org down on 2023-02-09 9:30AM

by Luís Tonicha

Hello all, My name is Luís and I'm part of the Deployments team. We will be migrating the TF LAVA instance from its current deployment to our new, Kubernetes-based platform, this Friday (09/02) at 9:30 (AM) UTC. This means that the tf.validation.linaro.org service will be down for the expected downtime of 2 to 3 hours. This is just a migration so no down/upgrades or configuration changes to LAVA will happen. Once the service is back online we'll announce it. Thanks for your collaboration. Regards, -- Luís Tonicha <luis.tonicha(a)linaro.org>

2 years, 1 month

1
2
0 0

[Maintenance] - urgent downtime for all Jenkins servers today 2024-01-29

by Kelley Spoon

Greetings All, The Jenkins servers on ci-staging.trustedfirmware ci.trustedfirmware.org and mbedtls.trustedfirmware.org will all be put into safe mode in a few moments in preparation for a shutdown around 6pm UTC today. The purpose of this is to perform an upgrade to the server to address a critical CVE discovered for the version of Jenkins running on all servers. Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years, 1 month

1
1
0 0

TF-A CI TuxSuite integration - official launch

by Paul Sokolovsky

Dear Colleagues, Thanks for your patience while the latest TuxSuite integration issues were worked our during recent weeks. Last of the systematic FVP integration issues were addressed end of last week, and since the last weekend, the system was in the "soft launch" mode to see if there're any remaining issues. Some intermittent issues were addressed during this week. As of now, all the main TF-A CI jobs should be converted, specifically: * Mainline daily builds * Mainline L1/L2 patch builds * LTS2.8 daily builds * LTS2.8 L1/L2 patch builds Smaller related issues should also be addressed by now (in particular https://linaro.atlassian.net/browse/TFC-573 "Make sure that Tux test failure leads to Jenkins build being set to Failed status" which a subject of explicit request). But thanks for understanding that we may still need some tweaks here and there during the coming week(s). We're launching integration with the highly scalable TuxSuite backend well in advance of the next mainline release, and just in time for the anticipated second LTS branch being forked. All that hopefully gives us enough time to test the new system further and make any following adjustments as needed. But my testing shows that the main "selling points" of the new system are already at play - higher scalability and improved stability due to avoidance of common testing bottlenecks we experienced during recent times as TF CI coverage and load grew in high tempo. Please feel free to share your feedback and/or questions. Thanks, Linaro TF CI maintenance team Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

2 years, 1 month

1
0
0 0

TF MISRA tool, ECLAIR, upgraded to 3.13

by Paul Sokolovsky

Hello, ECLAIR version 3.13 was released end of last year, and as of now, both TF-A and TF-M CI jobs are upgraded to this version. The jobs and reports seem to be well, but please let me know if you see any issues. Just in case, we keep docker images with the previous version around, so if there are issues or a need for detailed comparison of result of old vs new version, we can easily do that. Thanks, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

2 years, 2 months

1
0
0 0

[Maintenance] - ci.trustedfirmware.org and mbedtls.trustedfirmware.org down time on 2024-01-13

by Kelley Spoon

Hello All, The servers will stop processing jobs on 2024-1-20 at around 2:00pm UTC, as the servers will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins and some of its plugins to the latest LTS release. Start: 2024-1-13 03:00pm UTC End: 2024-1-13 07:00pm UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years, 2 months

1
1
0 0

Deploying OpenCI TuxSuite FVP integration

by Paul Sokolovsky

Hello, Overloading and scalability issues have been a known issue with OpenCI for a long time, usually happening around the release cycles due to elevated CI activity, but also occurring from time to time during normal workflow too. That's why last quarter we worked on an actionable plan to leverage TuxSuite, a recent Linaro technology for cloud-based building and testing, which proved itself well with other projects. For this initial pilot project, we're looking to route TF-A FVP tests (90+% of our test load) to TuxSuite away from LAVA, to alleviate load on existing build and physical device test infrastructure. It was actively worked on during this month, and I'm happy to to report that initial development and preliminary testing on staging show encouraging results. I'd like to perform larger-scale testing on staging yet, but otherwise think that we should be ready to deploy to production next. As it's a holiday season with not many working days left, I'm sending this a bit earlier to make sure it's not forgotten or comes as a surprise later. The plan is otherwise to test on staging today and/or over weekend, and then proceed with deployment and validating on production next week(s). Please let me know if you have questions or concerns. Happy holidays, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

2 years, 2 months

2
3
0 0

[Maintenance] - ci.staging.trustedfirmware.org down time 2023-12-20

by Kelley Spoon

Hello All, The server will stop processing jobs on 2023-12-20 at around 6:00pm UTC, as the server will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins and some of its plugins to the latest LTS release. Start: 2023-12-20 07:00pm UTC End: 2023-12-20 09:00 UTC If this upgrade remains stable in staging, we will plan to upgrade ci.trustedfirmware.org and mbedtls.trustedfirmware.org shortly after 2024-01-03. Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years, 3 months

1
1
0 0

[Maintenance] - ci.trustedfirmware.org down time 2023-10-13

by Kelley Spoon

Hello All, The jenkins server will stop processing jobs on 2023-10-13 at around 21:00 UTC, as the server will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins to address some minor security vulnerabilities and upgrade internal dependencies off of obsolete versions. This upgrade will be to the same version that's been running on staging for the past week and a half. Start: 2023-10-13 21:00 UTC Finish: 2023-10-13 23:00 UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years, 5 months

1
1
0 0

Switch to more consistent/user-friendly armclang toochain tarball naming scheme for Docker images

by Paul Sokolovsky

Hello, FYI, I'd like to switch to more natural armclang tarball naming scheme, as used to prepare TF docker build images: https://linaro.atlassian.net/browse/TFC-513 -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

2 years, 5 months

1
0
0 0

Update on OpenCI release readiness in regard to ability to sustain increased load

by Paul Sokolovsky

Hello, As you might know, preparing for the upcoming TF-A release, there was motion to test how the system behaves under the extra usually caused by the release work, to anticipate what to expect and possibly to make adjustments to improve situation comparing with the previous releases (were overloads were all but common). The testing started a few weeks ago by Joanna, joined last week by the OpenCI maintenance team, especially when "death spiral" system behavior was detected, familiar from the previous releases. This behavior was suspected to be caused, and have been confirmed by the following circumstances: 1. A patch is submitted and enabled for testing (AllowCI+2) which causes a large number of (LAVA) tests to fail due to timeout. 2. These tests keep FVP virtual devices in LAVA busy for much larger time than usual (~10x). 3. As there're many such tests, they block devices and cause LAVA queue to grow and bottleneck (400-500 tests waiting). 4. Jenkins jobs also retry failing LAVA tests number of times as a stopgap measure against non-deterministically and randomly failed tests. 5. As Jenkins jobs also have timeouts, waiting for queued/retries test results caused them timeout. 6. All these factors have a positive feedback effect on each, causing that "death spiral" effect when both Jenkins and LAVA were severely overloaded, while doing nothing useful (effectively, waiting). And the whole system was effectively deadlocked, where any new started job just kept waiting until its timeout to fail, requiring manual intervention to clear this state. The measure to address this situation were: 1. Decrease default LAVA test timeout to more reasonable values (better average value, while being ready to override it as needed for individual tests). 2. Decrease number of test failure retries. 3. Increase number of Jenkins and LAVA workers/devices/containers. The last Joanna's test after this change showed that the system no longer exhibits "death spiral" behavior under heavy, but realistic load. I also performed additional "extreme" test of running AllowCI+2 for multiple timeout-failing patches at once. I still was able to reproduce a situation when a Jenkins job timed out on its side, but at least there was no obvious "domino" effect to other jobs. All this work was tracked via https://linaro.atlassian.net/browse/TFC-498, which contain much more detail in it subtickets. This tasks is closed now, per above. Given that OpenCI is a complex and busy system, it is hard to be 100% that it was single underlying issue which caused the problems. So, if you see unexpected/problematic behavior, please open a TFC ticket (which is still the standard workflow to report and track issues). Thanks, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

2 years, 5 months

1
0
0 0

2026

2025

2024

2023

2022

Tf-openci