- Tf-openci - lists.trustedfirmware.org

Update on OpenCI release readiness in regard to ability to sustain increased load

by Paul Sokolovsky

Hello, As you might know, preparing for the upcoming TF-A release, there was motion to test how the system behaves under the extra usually caused by the release work, to anticipate what to expect and possibly to make adjustments to improve situation comparing with the previous releases (were overloads were all but common). The testing started a few weeks ago by Joanna, joined last week by the OpenCI maintenance team, especially when "death spiral" system behavior was detected, familiar from the previous releases. This behavior was suspected to be caused, and have been confirmed by the following circumstances: 1. A patch is submitted and enabled for testing (AllowCI+2) which causes a large number of (LAVA) tests to fail due to timeout. 2. These tests keep FVP virtual devices in LAVA busy for much larger time than usual (~10x). 3. As there're many such tests, they block devices and cause LAVA queue to grow and bottleneck (400-500 tests waiting). 4. Jenkins jobs also retry failing LAVA tests number of times as a stopgap measure against non-deterministically and randomly failed tests. 5. As Jenkins jobs also have timeouts, waiting for queued/retries test results caused them timeout. 6. All these factors have a positive feedback effect on each, causing that "death spiral" effect when both Jenkins and LAVA were severely overloaded, while doing nothing useful (effectively, waiting). And the whole system was effectively deadlocked, where any new started job just kept waiting until its timeout to fail, requiring manual intervention to clear this state. The measure to address this situation were: 1. Decrease default LAVA test timeout to more reasonable values (better average value, while being ready to override it as needed for individual tests). 2. Decrease number of test failure retries. 3. Increase number of Jenkins and LAVA workers/devices/containers. The last Joanna's test after this change showed that the system no longer exhibits "death spiral" behavior under heavy, but realistic load. I also performed additional "extreme" test of running AllowCI+2 for multiple timeout-failing patches at once. I still was able to reproduce a situation when a Jenkins job timed out on its side, but at least there was no obvious "domino" effect to other jobs. All this work was tracked via https://linaro.atlassian.net/browse/TFC-498, which contain much more detail in it subtickets. This tasks is closed now, per above. Given that OpenCI is a complex and busy system, it is hard to be 100% that it was single underlying issue which caused the problems. So, if you see unexpected/problematic behavior, please open a TFC ticket (which is still the standard workflow to report and track issues). Thanks, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

1 year, 8 months

1
0
0 0

TrustedFirmware MISRA Testing - September update

by Paul Sokolovsky

Hello, I would like to share some developments and updates regarding TrustedFirmware MISRA testing throughout September: 1. MISRA CI testing for TF-M was formally launched. That doesn't mean it runs to all its power yet, but the infrastructure is in place, and the next steps are for the TF-M team to see how it fits into their development workflow, and decide how to address identified MISRA issues - either record them as deviations or fix in the TF-M source code (likely combination of both). 2. One of the development done for the TF-M testing was implementation of the cumulative report across multiple configurations (vs myriad of individual per-configuration reports, which are hard to follow). This feature was already forward-ported to the TF-A "daily" build. It immediately made visible the fact that a MISRA mandatory rule violation crept into the codebase: https://ci-builds.trustedfirmware.org/static-files/llodfObQwsfBE_M8BN9W1URq… , select "Mandatory rules - violations" (note that the link will expire after some time). Further development plans are: 1. Cooperate with the TF-M team regarding MISRA rule, etc. configuration to get the reports into a shape useful for developers and contributors. 2. Forward-port cumulative report feature to the TF-A "delta" (i.e. patch) testing. These will be worked on starting from October, subject to other feature development and maintenance work. Thanks, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

1 year, 8 months

1
0
0 0

[Maintenance] - review.trustedfirmware.org down time 2023-10-03

by Kelley Spoon

Hello all, The gerrit server on review.trustedfirmware.org will be offline for a 2 hour maintenance window starting today (Oct 4, 2023) at 19:00 UTC. This downtime is needed in order to upgrade gerrit to the current version (3.8.1). Changelog is available at: -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 9 months

1
2
0 0

[Maintenance] - ci.staging.trustedfirmware.org down time 2023-10-03

by Kelley Spoon

Hello All, The jenkins server will stop processing jobs on 2023-10-03 at around 21:00 UTC, as the server will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins to address some minor security vulnerabilities and upgrade internal dependencies off of obsolete versions. Start: 2023-10-03 21:00 UTC Finish: 2023-10-03 23:00 UTC -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 9 months

1
1
0 0

[Maintenance] Emergency upgrade for all jenkins servers

by Kelley Spoon

Hello All, A recent CVE was published for the BlueOcean plugin being vulnerable to a low effort CSRF attack that could disclose github credentials[1] I have redeployed all Jenkins servers with an updated docker images that contains a fixed version of BlueOcean. The changes were minimal and should not be disruptive, but please let me know if you experience any problems. [1] https://www.jenkins.io/security/advisory/2023-08-16/#SECURITY-3116 Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

1 year, 10 months

1
0
0 0

[Maintenance] - ci.trustedfirmware.org down time 2023-06-16

by Kelley Spoon

Hello All, The server will stop processing jobs on 2023-06-16 at around 17:00 UTC, as the server will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins to match the same version that has been running in staging that addresses some potential security issues and formatting problems. This will be an upgrade to a minor point release and will not be introducing disruptive changes. Start: 2023-06-16 17:00 UTC End: 2023-06-16 19:00 UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years

1
0
0 0

[Maintenance] - ci.staging.trustedfirmware.org down time 2023-05-31

by Kelley Spoon

Hello All, The server will stop processing jobs on the 2023-05-31 at around 22:00 UTC, as the server will be put into "Shutdown mode". This downtime is needed in order to upgrade Jenkins and some of its plugins to address potential security issues. This will be an upgrade to a minor point release and will not be introducing disruptive changes. Start: 2023-05-31 23:00 UTC End: 2023-06-01 0:00 UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years, 1 month

1
1
0 0

[Maintenance] - TrustedFirmware Open CI down time 21/03/2023

by Kelley Spoon

Hello All, We are scheduling time for multiple components of the TrustedFirmware Open CI to be offline this Tuesday, March 21, 2023 in order to upgrade system components. Affected servers/services will be: - LAVA Main and its workers - Open CI Jenkins (ci.trustedfirmware.org) - FlexNet licensing This will also affect jobs running on the staging CI server. The maintenance window is scheduled to be 6 hours starting on 21/03/2023 around 11:00 UTC. Emails will be sent when the mainteance is complete and services are back online. Start: 21/03/2023 11:00 UTC End: 21/03/2023 17:00 UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years, 3 months

1
1
0 0

[Maintenance] - ci.staging.trustedfirmware.org down time 2023-03-11

by Kelley Spoon

Hello All, The server will offline to start a maintenance window for 2 hours on 2023-3-11 at 12:00PM UTC. Jenkins will be put into "Shutdown" and stop accepting jobs at 2023-3-11 11:00AM UTC. This downtime is required to update Jenkins as well as to introduce some plugins to add new functionality. An email will be sent after the maintenance has been completed and Jenkins has resumed normal operations. Start: 2023-03-11 12:00 UTC End: 2023-03-11 14:00 UTC Regards, -- Kelley Spoon <kelley.spoon(a)linaro.org>

2 years, 3 months

1
1
0 0

[Maintenance] - tf.validation.linaro.org

by Benjamin Copeland

Hello All, As announced in yesterdays meeting the LAVA Lab team will be working on migrating the LAVA instance to the cloud from tf-master on Thursday 11am UTC. This is our first step to the upgrade, we will have some future announcements coming in when we upgrade the workers and do some Jenkins updates. Start: 23/02/23 11:00UTC End: 23/02/23 12:00UTC It shouldn't take one hour, but the window is there just in case of any issues. Regards Ben

2 years, 4 months

1
0
0 0

2025

2024

2023

2022

Tf-openci