TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Cheers! Okash
[1] https://lists.trustedfirmware.org/archives/search?mlist=tf-a%40lists.trusted... [2] https://www.trustedfirmware.org/docs/TF-A-LTS.pdf [3] https://lists.trustedfirmware.org/archives/list/tf-a@lists.trustedfirmware.o... [4] https://www.trustedfirmware.org/docs/TForg_LTS_proposal.pdf
One thing to consider for the LTS technical proposal is a release is made up of multiple repositories and other collateral (eg Readthedocs).
Take the v2.7 release https://lists.trustedfirmware.org/archives/list/tf-a@lists.trustedfirmware.o...
As can be seen its made up of more than TF-A and TF-A Tests (TFTF). It also incorporates Hafnium and TF-A OpenCI Scripts. Not tagged is the TF-A OpenCI Jobs (Jenkin Jobs). All generally tested and released together for the release and tagged. Other collateral includes the binaries used during testing (eg compiler versions and other binaries used during testing like FVP’s etc) that make up the supply chain of a release. You may want to consider if this proposal should incorporate all or some of these in the LTS proposal or just stick with the source code sub-components of a release.
Joanna
From: Okash Khawaja via TSC tsc@lists.trustedfirmware.org Date: Monday, 8 August 2022 at 18:50 To: tf-a@lists.trustedfirmware.org tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org tsc@lists.trustedfirmware.org, Varun Wadekar vwadekar@nvidia.com Subject: [TF-TSC] Finally, a TF-A LTS Proposal! TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Cheers! Okash
[1] https://lists.trustedfirmware.org/archives/search?mlist=tf-a%40lists.trusted... [2] https://www.trustedfirmware.org/docs/TF-A-LTS.pdf [3] https://lists.trustedfirmware.org/archives/list/tf-a@lists.trustedfirmware.o... [4] https://www.trustedfirmware.org/docs/TForg_LTS_proposal.pdf
Thanks Joanna. We discussed Hafnium addition in the initial release, but thought that can be an incremental update in the next LTS release.
We would need inputs on the tf.org infrastructure and the CI/CD flow. If you think TF-A OpenCI Scripts should be part of the LTS release, then we can include the project.
The aim of the proposal is to start small to solidify the flow and add more as we move forward.
From: Joanna Farley Joanna.Farley@arm.com Sent: Monday, 8 August 2022 1:55 PM To: Okash Khawaja okash@google.com; tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com Subject: Re: [TF-TSC] Finally, a TF-A LTS Proposal!
External email: Use caution opening links or attachments
One thing to consider for the LTS technical proposal is a release is made up of multiple repositories and other collateral (eg Readthedocs).
Take the v2.7 release https://lists.trustedfirmware.org/archives/list/tf-a@lists.trustedfirmware.o...
As can be seen its made up of more than TF-A and TF-A Tests (TFTF). It also incorporates Hafnium and TF-A OpenCI Scripts. Not tagged is the TF-A OpenCI Jobs (Jenkin Jobs). All generally tested and released together for the release and tagged. Other collateral includes the binaries used during testing (eg compiler versions and other binaries used during testing like FVP's etc) that make up the supply chain of a release. You may want to consider if this proposal should incorporate all or some of these in the LTS proposal or just stick with the source code sub-components of a release.
Joanna
From: Okash Khawaja via TSC <tsc@lists.trustedfirmware.orgmailto:tsc@lists.trustedfirmware.org> Date: Monday, 8 August 2022 at 18:50 To: tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org <tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org> Cc: tsc@lists.trustedfirmware.orgmailto:tsc@lists.trustedfirmware.org <tsc@lists.trustedfirmware.orgmailto:tsc@lists.trustedfirmware.org>, Varun Wadekar <vwadekar@nvidia.commailto:vwadekar@nvidia.com> Subject: [TF-TSC] Finally, a TF-A LTS Proposal! TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Cheers! Okash
[1] https://lists.trustedfirmware.org/archives/search?mlist=tf-a%40lists.trusted...https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.trustedfirmware.org%2Farchives%2Fsearch%3Fmlist%3Dtf-a%2540lists.trustedfirmware.org%26q%3D%255BTF-A%255D%2BATF%2BLTS%2Bversion&data=05%7C01%7Cvwadekar%40nvidia.com%7C5980beb3d7ab468f5e4b08da79805220%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637955889898278635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yRtvFLqvEpAS%2BuMJPYf28kf3kEOJjDMVxciOac81p%2FU%3D&reserved=0 [2] https://www.trustedfirmware.org/docs/TF-A-LTS.pdfhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.trustedfirmware.org%2Fdocs%2FTF-A-LTS.pdf&data=05%7C01%7Cvwadekar%40nvidia.com%7C5980beb3d7ab468f5e4b08da79805220%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637955889898434869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5A66vxdeeKEH%2BROI1g2CRlTDll%2FBxJiEHXn546xoe3Y%3D&reserved=0 [3] https://lists.trustedfirmware.org/archives/list/tf-a@lists.trustedfirmware.o...https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.trustedfirmware.org%2Farchives%2Flist%2Ftf-a%40lists.trustedfirmware.org%2Fthread%2FZZ2DDIW2DS4B7QYHYOL7XE4IPUW7LUNT%2F&data=05%7C01%7Cvwadekar%40nvidia.com%7C5980beb3d7ab468f5e4b08da79805220%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637955889898434869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yXy0qesKEWdlDszqv%2BZzJuiZqm5Pp%2BGsNzhcc68mg6A%3D&reserved=0 [4] https://www.trustedfirmware.org/docs/TForg_LTS_proposal.pdfhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.trustedfirmware.org%2Fdocs%2FTForg_LTS_proposal.pdf&data=05%7C01%7Cvwadekar%40nvidia.com%7C5980beb3d7ab468f5e4b08da79805220%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637955889898434869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=L3DjNSfDtyFEPueL%2FWUfYK7wdpTeg6MpThDxvxq7rpk%3D&reserved=0
Hi Okash, Varun, all,
On Mon, Aug 08, 2022 at 06:48:42PM +0100, Okash Khawaja via TSC wrote:
TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Thanks for putting this together, much appreciated. Some comments from me/Linaro.
Criteria: -------- - "large parts of the process can be automated" I'm not sure I'm fully convinced about that, but if CI/CD/testing have been configured to have good support for LTS-branches, then the actual and manual work is mostly doing the actual backporting. Aiming for an automated process is a good goal.
- "No features will be backported" As for the criteria I believe your suggestion makes a lot of sense. Even though I do agree with the "No features will be backported", I'm pretty sure that from time to time there will be fixes that for some reason need to pull in features as well, mainly as a dependency. Perhaps we could change the sentence slightly to indicate that we generally don't want to backport features, but if a fix depends or is dependant on a feature, then in some cases we might consider backporting features as well. Case by case? TF-TSC people could decide?
Lifetime and frequency: ---------------------- I support your suggestions and motivations.
Testing Criteria / 4.Platform tests: ----------------------------------- Although I support this, I can see foresee that this most likely will be challenging. I'm talking from my experience as an OP-TEE maintainer. As the amount of platforms supported grew, it became harder and harder to get the platform maintainers finding time and helping out with testing release candidates on their own devices/platforms. We have tried a couple of different things, like using stop lights in the docs, keeping tested platforms "green", the ones that hasn't been tested recently "orange" and the rest "red". However, we abandoned that, since it was a lot of work to keep track of it. Instead we simply just keep track of the tested platforms in the git release tag. I.e., we list all platforms tested both by the core maintainers as well as external platform maintainers. Sub-optimal, but keeps a track record and keeps things moving. Here I'm mainly talking about platforms found upstream. In your suggestion you're talking about platforms that are not maintained upstream, which in my opinion makes things even more complicated.
In short, I do support something like this, but I think there needs to be some kind of deadline for downstream testing. If not met, then that platform shouldn't be considered as up-to-date for a LTS. I think this is what you also had in mind with "a pre-defined window".
TFTF branching and others gits: ------------------------------ I agree with the need to "LTS tag" TFTF and potentially other projects as well. I.e., as Joanna indicated, enabling LTS in TF-A might force us to add LTS branches in the other TF.org gits as well. I've touched upon it before, but maybe time for putting together TF.org build(s) that builds (and tests) all projects under the TrustedFirmware umbrella?
Example timeline: ---------------- - "Note that here we don’t allow the extra testing and debugging time that we had between Nov 2022 and early Feb 2023. This is because there isn’t as much to test and debug as an annual LTS release has."
I think this is related to my comment further up. If the "large part of the process can be automated" holds up, then I'd guess that the same kind of testing could (and should) be done.
Maintainership: -------------- In short LTS doesn't come for free, so in some way it needs to be funded. Carving out LTS work from existing work as a maintainer can be challenging and a lot of maintainers spend time on other "internal" tasks and use cases as well.
- "A day in the life of a maintainer" I guess this sections justify my doubts that "large parts of the process can be automated".
Execution Plan: -------------- +1
Future plans: ------------ Agree, it's hard to make a bulletproof strategy, but having a strategy capturing most of the things is a lot better than having no strategy at all.
Again, thanks for putting together the proposal, it's a good proposal.
Cheers! Okash
[1] https://lists.trustedfirmware.org/archives/search?mlist=tf-a%40lists.trusted... [2] https://www.trustedfirmware.org/docs/TF-A-LTS.pdf [3] https://lists.trustedfirmware.org/archives/list/tf-a@lists.trustedfirmware.o... [4] https://www.trustedfirmware.org/docs/TForg_LTS_proposal.pdf
-- TSC mailing list -- tsc@lists.trustedfirmware.org To unsubscribe send an email to tsc-leave@lists.trustedfirmware.org
// Regards, Joakim
Hi Joakim,
Thanks for the feedback. My comments below.
I'm not sure I'm fully convinced about that, but if CI/CD/testing have been configured to have good support for LTS-branches, then the actual and manual work is mostly doing the actual backporting. Aiming for an automated process is a good goal.
[VW] Yes, we aim to enable the CI/CD/Testing process for LTS branches. The aim is also for an automated process to take a first stab at the backporting. Only if it fails, will we need manual intervention.
I'm pretty sure that from time to time there will be fixes that for some reason need to pull in features as well, mainly as a dependency
[VW] Do you have an example? But in general, we should look at each situation case by case. Although the guidance has been drawn for the first release, we are aware that the criteria will and has to evolve over time.
I think there needs to be some kind of deadline for downstream testing. If not met, then that platform shouldn't be considered as up-to-date for a LTS. I think this is what you also had in mind with "a pre-defined window".
[VW] That's right. We want to introduce a deadline to avoid the problem you faced. We want to increase interest and support for all platforms, but don't want to penalize some due to slow response from others.
I've touched upon it before, but maybe time for putting together TF.org build(s) that builds (and tests) all projects under the TrustedFirmware umbrella?
[VW] I'm afraid, I don't understand. Can you please elaborate? Are you proposing something on the line of the TrustedSubstrate project?
If the "large part of the process can be automated" holds up, then I'd guess that the same kind of testing could (and should) be done.
[VW] That's the end goal - release LTS and normal release at the same time.
Thanks, Varun
-----Original Message----- From: Joakim Bech joakim.bech@linaro.org Sent: Thursday, 11 August 2022 10:37 AM To: Okash Khawaja okash@google.com Cc: tf-a@lists.trustedfirmware.org; tsc@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com Subject: Re: [TF-TSC] Finally, a TF-A LTS Proposal!
External email: Use caution opening links or attachments
Hi Okash, Varun, all,
On Mon, Aug 08, 2022 at 06:48:42PM +0100, Okash Khawaja via TSC wrote:
TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Thanks for putting this together, much appreciated. Some comments from me/Linaro.
Criteria: -------- - "large parts of the process can be automated" I'm not sure I'm fully convinced about that, but if CI/CD/testing have been configured to have good support for LTS-branches, then the actual and manual work is mostly doing the actual backporting. Aiming for an automated process is a good goal.
- "No features will be backported" As for the criteria I believe your suggestion makes a lot of sense. Even though I do agree with the "No features will be backported", I'm pretty sure that from time to time there will be fixes that for some reason need to pull in features as well, mainly as a dependency. Perhaps we could change the sentence slightly to indicate that we generally don't want to backport features, but if a fix depends or is dependant on a feature, then in some cases we might consider backporting features as well. Case by case? TF-TSC people could decide?
Lifetime and frequency: ---------------------- I support your suggestions and motivations.
Testing Criteria / 4.Platform tests: ----------------------------------- Although I support this, I can see foresee that this most likely will be challenging. I'm talking from my experience as an OP-TEE maintainer. As the amount of platforms supported grew, it became harder and harder to get the platform maintainers finding time and helping out with testing release candidates on their own devices/platforms. We have tried a couple of different things, like using stop lights in the docs, keeping tested platforms "green", the ones that hasn't been tested recently "orange" and the rest "red". However, we abandoned that, since it was a lot of work to keep track of it. Instead we simply just keep track of the tested platforms in the git release tag. I.e., we list all platforms tested both by the core maintainers as well as external platform maintainers. Sub-optimal, but keeps a track record and keeps things moving. Here I'm mainly talking about platforms found upstream. In your suggestion you're talking about platforms that are not maintained upstream, which in my opinion makes things even more complicated.
In short, I do support something like this, but I think there needs to be some kind of deadline for downstream testing. If not met, then that platform shouldn't be considered as up-to-date for a LTS. I think this is what you also had in mind with "a pre-defined window".
TFTF branching and others gits: ------------------------------ I agree with the need to "LTS tag" TFTF and potentially other projects as well. I.e., as Joanna indicated, enabling LTS in TF-A might force us to add LTS branches in the other TF.org gits as well. I've touched upon it before, but maybe time for putting together TF.org build(s) that builds (and tests) all projects under the TrustedFirmware umbrella?
Example timeline: ---------------- - "Note that here we don't allow the extra testing and debugging time that we had between Nov 2022 and early Feb 2023. This is because there isn't as much to test and debug as an annual LTS release has."
I think this is related to my comment further up. If the "large part of the process can be automated" holds up, then I'd guess that the same kind of testing could (and should) be done.
Maintainership: -------------- In short LTS doesn't come for free, so in some way it needs to be funded. Carving out LTS work from existing work as a maintainer can be challenging and a lot of maintainers spend time on other "internal" tasks and use cases as well.
- "A day in the life of a maintainer" I guess this sections justify my doubts that "large parts of the process can be automated".
Execution Plan: -------------- +1
Future plans: ------------ Agree, it's hard to make a bulletproof strategy, but having a strategy capturing most of the things is a lot better than having no strategy at all.
Again, thanks for putting together the proposal, it's a good proposal.
Cheers! Okash
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.trustedfirmware.org%2Farchives%2Fsearch%3Fmlist%3Dtf-a%2540lists.tru stedfirmware.org%26q%3D%255BTF-A%255D%2BATF%2BLTS%2Bversion&data=0 5%7C01%7Cvwadekar%40nvidia.com%7C2e60e0205ed040554ad408da7b7d001e%7C43 083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637958074486673316%7CUnknown% 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX VCI6Mn0%3D%7C3000%7C%7C%7C&sdata=z7%2FJNwIGWVAXBNXLNPr5ZGSo558XlB% 2FiEW4UIPHR9L4%3D&reserved=0 [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. trustedfirmware.org%2Fdocs%2FTF-A-LTS.pdf&data=05%7C01%7Cvwadekar% 40nvidia.com%7C2e60e0205ed040554ad408da7b7d001e%7C43083d15727340c1b7db 39efd9ccc17a%7C0%7C0%7C637958074486673316%7CUnknown%7CTWFpbGZsb3d8eyJW IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% 7C%7C%7C&sdata=42hoFTBXh702PcZya9ZcMYWL8V2CG4Y5LUGmf3pjQc4%3D& reserved=0 [3] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.trustedfirmware.org%2Farchives%2Flist%2Ftf-a%40lists.trustedfirmware .org%2Fthread%2FZZ2DDIW2DS4B7QYHYOL7XE4IPUW7LUNT%2F&data=05%7C01%7 Cvwadekar%40nvidia.com%7C2e60e0205ed040554ad408da7b7d001e%7C43083d1572 7340c1b7db39efd9ccc17a%7C0%7C0%7C637958074486673316%7CUnknown%7CTWFpbG Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% 3D%7C3000%7C%7C%7C&sdata=QjLiQizT6m3VgqblAudtwA%2FQAq%2BcU4vdWJiBL VWTz00%3D&reserved=0 [4] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. trustedfirmware.org%2Fdocs%2FTForg_LTS_proposal.pdf&data=05%7C01%7 Cvwadekar%40nvidia.com%7C2e60e0205ed040554ad408da7b7d001e%7C43083d1572 7340c1b7db39efd9ccc17a%7C0%7C0%7C637958074486673316%7CUnknown%7CTWFpbG Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% 3D%7C3000%7C%7C%7C&sdata=g2kYD%2F%2FxStAkqxIppPYjp8rUF0vBF6T2zz9ZW USNSEE%3D&reserved=0
-- TSC mailing list -- tsc@lists.trustedfirmware.org To unsubscribe send an email to tsc-leave@lists.trustedfirmware.org
// Regards, Joakim
Hi Okash/Varun/all
Thanks for this. Some comments from me:
* Criteria: I agree with Joakim that in some cases, a feature may need to be backported as a dependency, although I think this can be mainly decided by the LTS maintainer(s) on a case by case basis. TSC should only be needed where there's controversy/disagreement. There's no mention of normal (i.e. non-security) bugs in generic code. Presumably these don't meet the criteria (?), but it's better to be explicit.
* Frequency: Yearly in November makes sense but if they live for 5/7 years that obviously means 5/7 concurrent branches to maintain, which will make automation even more important.
* Regarding automation, clearly all the CI needs to be automated from the start. It may be possible to manually backport to start with but even this will need to be mostly automated if there are many concurrent branches. I know with the Linux LTS branches that patches are tagged for backporting in the mainline commit messages, and scripts automatically backport to the LTS branches. When patches fail to cleanly backport, it's up to the LTS community to address as it's not scalable for the LTS maintainer(s) to fix all problems. We could consider copying this model (even the scripts) eventually. You mention tagging patches on mainline instead, which could work but I guess could result in tag proliferation.
* Testing criteria: +1 on Joakim's comment about downstream testing not gating LTS patch merging. I’d also say that platforms can only be tested in the LTS CI if they are also tested in the mainline CI.
* Test-and-debug period: "it is expected that more testing and debugging will be done on the November release of TF-A". The testing criteria in the previous section suggest LTS tests are a subset of mainline testing, so in theory the LTS release could be made at the same time as main release. I guess you mean there needs to be more downstream testing of the mainline release before the first LTS release is made. That's fine but this should be made clear and there should be a fixed window so that downstream testing doesn't gate the LTS release.
* "[LTS Maintainer to] Monitor the main branch to identify candidate patches for the LTS branches". This might work to start with but I don't think is very scalable. I think it's reasonable to make all TF-A contributors responsible for identifying such patches at the time they created, with the LTS maintainer acting as backup when things are missed.
* "After reviews are complete, merge the patch and bump the minor version, if required". This suggest no downstream testing is required for minor LTS versions - is that right? I would have thought a small window is required. Or can we assume that further patches/releases are made if there are problems?
* "What happens when a bug fix applies just to a LTS branch and not to the master branch?". If a bug is only reproducible on the LTS branch, then it seems to me this is a reasonable exception to the "only backport from mainline" rule.
* "When testing a backported patch, what if one of the partners needs more time while the patch fix is time-critical and hence slowing other partners?". I would say there should be a fixed window. If a partner subsequently finds a problem, another minor release can be made.
* I suspect a new TF-A LTS mailing list will be required for this (TBD).
* Maintainership options: This is going to take more discussion, but my personal view is the current set of maintainers/contributors does not closely intersect with those interested in consuming an LTS, so it's wouldn't be fair for the current maintainers/contributors to take the whole burden of LTS maintainership.
Regards
Dan.
-----Original Message----- From: Okash Khawaja via TSC tsc@lists.trustedfirmware.org Sent: 08 August 2022 18:49 To: tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com Subject: [TF-TSC] Finally, a TF-A LTS Proposal!
TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Cheers! Okash
[1] https://lists.trustedfirmware.org/archives/search?mlist=tf- a%40lists.trustedfirmware.org&q=%5BTF-A%5D+ATF+LTS+version [2] https://www.trustedfirmware.org/docs/TF-A-LTS.pdf [3] https://lists.trustedfirmware.org/archives/list/tf- a@lists.trustedfirmware.org/thread/ZZ2DDIW2DS4B7QYHYOL7XE4IPUW7LUNT/ [4] https://www.trustedfirmware.org/docs/TForg_LTS_proposal.pdf
Hi,
Thanks for the feedback. My comments below.
There's no mention of normal (i.e. non-security) bugs in generic code. Presumably these don't meet the criteria (?), but it's better to be explicit.
[VW] Makes sense. The intention is to focus on security issues, but I agree that we should look at such bugs on a case-by-case basis. We will try to include this in the guidance.
I know with the Linux LTS branches that patches are tagged for backporting in the mainline commit messages, and scripts automatically backport to the LTS branches. When patches fail to cleanly backport, it's up to the LTS community to address as it's not scalable for the LTS maintainer(s) to fix all problems. We could consider copying this model (even the scripts) eventually.
[VW] That is a good idea. I agree we should try to replicate mechanisms that already work.
I guess you mean there needs to be more downstream testing of the mainline release before the first LTS release is made. That's fine but this should be made clear and there should be a fixed window so that downstream testing doesn't gate the LTS release.
[VW] The intention was to allow platforms more testing time. We have tried to provide a deadline but will try to make it clear.
I think it's reasonable to make all TF-A contributors responsible for identifying such patches at the time they created, with the LTS maintainer acting as backup when things are missed.
[VW] This is what I had in mind at the start and want to aim for. But, want to catch scenarios where contributors miss this. So, we want a reviewer list that never misses such a patch and then educate the contributors to get to the end goal.
"After reviews are complete, merge the patch and bump the minor version, if required". This suggest no downstream testing is required for minor LTS versions - is that right? I would have thought a small window is required. Or can we assume that further patches/releases are made if there are problems?
[VW] The intent is to complete the testing _before_ the patch is merged to the branch. This testing again has a deadline to avoid slowing down the release process.
"What happens when a bug fix applies just to a LTS branch and not to the master branch?". If a bug is only reproducible on the LTS branch, then it seems to me this is a reasonable exception to the "only backport from mainline" rule.
[VW] Agreed.
"When testing a backported patch, what if one of the partners needs more time while the patch fix is time-critical and hence slowing other partners?". I would say there should be a fixed window. If a partner subsequently finds a problem, another minor release can be made.
[VW] Agreed.
I suspect a new TF-A LTS mailing list will be required for this
[VW] Probably. Would like to address this in the Tech forum.
This is going to take more discussion, but my personal view is the current set of maintainers/contributors does not closely intersect with those interested in consuming an LTS, so it's wouldn't be fair for the current maintainers/contributors to take the whole burden of LTS maintainership.
[VW] I understand. In the absence of a formal group, the easiest was forward was to request the current maintainers. We should discuss more in the Tech forum.
-Varun
-----Original Message----- From: Dan Handley Dan.Handley@arm.com Sent: Thursday, 11 August 2022 12:12 PM To: Okash Khawaja okash@google.com; tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com Subject: RE: [TF-TSC] Finally, a TF-A LTS Proposal!
External email: Use caution opening links or attachments
Hi Okash/Varun/all
Thanks for this. Some comments from me:
* Criteria: I agree with Joakim that in some cases, a feature may need to be backported as a dependency, although I think this can be mainly decided by the LTS maintainer(s) on a case by case basis. TSC should only be needed where there's controversy/disagreement. There's no mention of normal (i.e. non-security) bugs in generic code. Presumably these don't meet the criteria (?), but it's better to be explicit.
* Frequency: Yearly in November makes sense but if they live for 5/7 years that obviously means 5/7 concurrent branches to maintain, which will make automation even more important.
* Regarding automation, clearly all the CI needs to be automated from the start. It may be possible to manually backport to start with but even this will need to be mostly automated if there are many concurrent branches. I know with the Linux LTS branches that patches are tagged for backporting in the mainline commit messages, and scripts automatically backport to the LTS branches. When patches fail to cleanly backport, it's up to the LTS community to address as it's not scalable for the LTS maintainer(s) to fix all problems. We could consider copying this model (even the scripts) eventually. You mention tagging patches on mainline instead, which could work but I guess could result in tag proliferation.
* Testing criteria: +1 on Joakim's comment about downstream testing not gating LTS patch merging. I'd also say that platforms can only be tested in the LTS CI if they are also tested in the mainline CI.
* Test-and-debug period: "it is expected that more testing and debugging will be done on the November release of TF-A". The testing criteria in the previous section suggest LTS tests are a subset of mainline testing, so in theory the LTS release could be made at the same time as main release. I guess you mean there needs to be more downstream testing of the mainline release before the first LTS release is made. That's fine but this should be made clear and there should be a fixed window so that downstream testing doesn't gate the LTS release.
* "[LTS Maintainer to] Monitor the main branch to identify candidate patches for the LTS branches". This might work to start with but I don't think is very scalable. I think it's reasonable to make all TF-A contributors responsible for identifying such patches at the time they created, with the LTS maintainer acting as backup when things are missed.
* "After reviews are complete, merge the patch and bump the minor version, if required". This suggest no downstream testing is required for minor LTS versions - is that right? I would have thought a small window is required. Or can we assume that further patches/releases are made if there are problems?
* "What happens when a bug fix applies just to a LTS branch and not to the master branch?". If a bug is only reproducible on the LTS branch, then it seems to me this is a reasonable exception to the "only backport from mainline" rule.
* "When testing a backported patch, what if one of the partners needs more time while the patch fix is time-critical and hence slowing other partners?". I would say there should be a fixed window. If a partner subsequently finds a problem, another minor release can be made.
* I suspect a new TF-A LTS mailing list will be required for this (TBD).
* Maintainership options: This is going to take more discussion, but my personal view is the current set of maintainers/contributors does not closely intersect with those interested in consuming an LTS, so it's wouldn't be fair for the current maintainers/contributors to take the whole burden of LTS maintainership.
Regards
Dan.
-----Original Message----- From: Okash Khawaja via TSC tsc@lists.trustedfirmware.org Sent: 08 August 2022 18:49 To: tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com Subject: [TF-TSC] Finally, a TF-A LTS Proposal!
TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Cheers! Okash
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.trustedfirmware.org%2Farchives%2Fsearch%3Fmlist%3Dtf-&data=05%7C 01%7Cvwadekar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTW FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0%3D%7C3000%7C%7C%7C&sdata=37mdLWDEDIRwAnsxo9c7bu2KwDq9oX%2B3Y4j %2BNiHzI%2Bc%3D&reserved=0 a%40lists.trustedfirmware.org&q=%5BTF-A%5D+ATF+LTS+version [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. trustedfirmware.org%2Fdocs%2FTF-A-LTS.pdf&data=05%7C01%7Cvwadekar% 40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d15727340c1b7db 39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbGZsb3d8eyJW IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% 7C%7C%7C&sdata=05dYPwDmH7OZMjMVb0AtN9eSJR8R4InaZ5ANo8qqqUA%3D& reserved=0 [3] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.trustedfirmware.org%2Farchives%2Flist%2Ftf-&data=05%7C01%7Cvwade kar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d15727340c1 b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbGZsb3d8 eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3 000%7C%7C%7C&sdata=jjsY%2B9EEZQiTqg5o9RLucSB5WzwsheWOOUIzXz5PRUQ%3 D&reserved=0 a@lists.trustedfirmware.org/thread/ZZ2DDIW2DS4B7QYHYOL7XE4IPUW7LUNT/ [4] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. trustedfirmware.org%2Fdocs%2FTForg_LTS_proposal.pdf&data=05%7C01%7 Cvwadekar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d1572 7340c1b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbG Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% 3D%7C3000%7C%7C%7C&sdata=q1Pk6ucv8RA7g3Yj59CMkrb2ZRY1abNHSiCSHJBOn hg%3D&reserved=0
Hello,
I have uploaded the proposal doc as a pdf to the LTS Phabricator page [1]. Please review and provide feedback.
Thanks.
[1] https://developer.trustedfirmware.org/w/tf_a/lts_proposal/
-----Original Message----- From: Varun Wadekar Sent: Tuesday, 6 September 2022 1:24 PM To: Dan Handley Dan.Handley@arm.com; Okash Khawaja okash@google.com; tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org Subject: RE: [TF-TSC] Finally, a TF-A LTS Proposal!
Hi,
Thanks for the feedback. My comments below.
There's no mention of normal (i.e. non-security) bugs in generic code. Presumably these don't meet the criteria (?), but it's better to be explicit.
[VW] Makes sense. The intention is to focus on security issues, but I agree that we should look at such bugs on a case-by-case basis. We will try to include this in the guidance.
I know with the Linux LTS branches that patches are tagged for backporting in the mainline commit messages, and scripts automatically backport to the LTS branches. When patches fail to cleanly backport, it's up to the LTS community to address as it's not scalable for the LTS maintainer(s) to fix all problems. We could consider copying this model (even the scripts) eventually.
[VW] That is a good idea. I agree we should try to replicate mechanisms that already work.
I guess you mean there needs to be more downstream testing of the mainline release before the first LTS release is made. That's fine but this should be made clear and there should be a fixed window so that downstream testing doesn't gate the LTS release.
[VW] The intention was to allow platforms more testing time. We have tried to provide a deadline but will try to make it clear.
I think it's reasonable to make all TF-A contributors responsible for identifying such patches at the time they created, with the LTS maintainer acting as backup when things are missed.
[VW] This is what I had in mind at the start and want to aim for. But, want to catch scenarios where contributors miss this. So, we want a reviewer list that never misses such a patch and then educate the contributors to get to the end goal.
"After reviews are complete, merge the patch and bump the minor version, if required". This suggest no downstream testing is required for minor LTS versions - is that right? I would have thought a small window is required. Or can we assume that further patches/releases are made if there are problems?
[VW] The intent is to complete the testing _before_ the patch is merged to the branch. This testing again has a deadline to avoid slowing down the release process.
"What happens when a bug fix applies just to a LTS branch and not to the master branch?". If a bug is only reproducible on the LTS branch, then it seems to me this is a reasonable exception to the "only backport from mainline" rule.
[VW] Agreed.
"When testing a backported patch, what if one of the partners needs more time while the patch fix is time-critical and hence slowing other partners?". I would say there should be a fixed window. If a partner subsequently finds a problem, another minor release can be made.
[VW] Agreed.
I suspect a new TF-A LTS mailing list will be required for this
[VW] Probably. Would like to address this in the Tech forum.
This is going to take more discussion, but my personal view is the current set of maintainers/contributors does not closely intersect with those interested in consuming an LTS, so it's wouldn't be fair for the current maintainers/contributors to take the whole burden of LTS maintainership.
[VW] I understand. In the absence of a formal group, the easiest was forward was to request the current maintainers. We should discuss more in the Tech forum.
-Varun
-----Original Message----- From: Dan Handley Dan.Handley@arm.com Sent: Thursday, 11 August 2022 12:12 PM To: Okash Khawaja okash@google.com; tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com Subject: RE: [TF-TSC] Finally, a TF-A LTS Proposal!
External email: Use caution opening links or attachments
Hi Okash/Varun/all
Thanks for this. Some comments from me:
* Criteria: I agree with Joakim that in some cases, a feature may need to be backported as a dependency, although I think this can be mainly decided by the LTS maintainer(s) on a case by case basis. TSC should only be needed where there's controversy/disagreement. There's no mention of normal (i.e. non-security) bugs in generic code. Presumably these don't meet the criteria (?), but it's better to be explicit.
* Frequency: Yearly in November makes sense but if they live for 5/7 years that obviously means 5/7 concurrent branches to maintain, which will make automation even more important.
* Regarding automation, clearly all the CI needs to be automated from the start. It may be possible to manually backport to start with but even this will need to be mostly automated if there are many concurrent branches. I know with the Linux LTS branches that patches are tagged for backporting in the mainline commit messages, and scripts automatically backport to the LTS branches. When patches fail to cleanly backport, it's up to the LTS community to address as it's not scalable for the LTS maintainer(s) to fix all problems. We could consider copying this model (even the scripts) eventually. You mention tagging patches on mainline instead, which could work but I guess could result in tag proliferation.
* Testing criteria: +1 on Joakim's comment about downstream testing not gating LTS patch merging. I'd also say that platforms can only be tested in the LTS CI if they are also tested in the mainline CI.
* Test-and-debug period: "it is expected that more testing and debugging will be done on the November release of TF-A". The testing criteria in the previous section suggest LTS tests are a subset of mainline testing, so in theory the LTS release could be made at the same time as main release. I guess you mean there needs to be more downstream testing of the mainline release before the first LTS release is made. That's fine but this should be made clear and there should be a fixed window so that downstream testing doesn't gate the LTS release.
* "[LTS Maintainer to] Monitor the main branch to identify candidate patches for the LTS branches". This might work to start with but I don't think is very scalable. I think it's reasonable to make all TF-A contributors responsible for identifying such patches at the time they created, with the LTS maintainer acting as backup when things are missed.
* "After reviews are complete, merge the patch and bump the minor version, if required". This suggest no downstream testing is required for minor LTS versions - is that right? I would have thought a small window is required. Or can we assume that further patches/releases are made if there are problems?
* "What happens when a bug fix applies just to a LTS branch and not to the master branch?". If a bug is only reproducible on the LTS branch, then it seems to me this is a reasonable exception to the "only backport from mainline" rule.
* "When testing a backported patch, what if one of the partners needs more time while the patch fix is time-critical and hence slowing other partners?". I would say there should be a fixed window. If a partner subsequently finds a problem, another minor release can be made.
* I suspect a new TF-A LTS mailing list will be required for this (TBD).
* Maintainership options: This is going to take more discussion, but my personal view is the current set of maintainers/contributors does not closely intersect with those interested in consuming an LTS, so it's wouldn't be fair for the current maintainers/contributors to take the whole burden of LTS maintainership.
Regards
Dan.
-----Original Message----- From: Okash Khawaja via TSC tsc@lists.trustedfirmware.org Sent: 08 August 2022 18:49 To: tf-a@lists.trustedfirmware.org Cc: tsc@lists.trustedfirmware.org; Varun Wadekar vwadekar@nvidia.com Subject: [TF-TSC] Finally, a TF-A LTS Proposal!
TL;DR: Attached is the TF-A LTS proposal doc. Please review and share your thoughts.
Hello everyone,
Long term support for TF-A has a long history. As a community we have flirted with it for a while, and like any flirtation, it started with gossip -- in our case, over mailing list [1] -- more than two years ago. Then Varun Wadekar did a wonderful tech forum presentation [2] which demonstrated interest in the topic and raised interesting questions about LTS.
After some time, the affair lost headline status, only to be revived again [3], earlier this year. This time however, there is a formal proposal (no pun intended). After the mailing list discussion and feedback from TF-A tech forum [4], we have put together a draft which attempts to give a concrete idea about what the TF-A LTS will look like.
Please spare some time to read through it and share your feedback. The plan is to put LTS into action this November.
Cheers! Okash
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.trustedfirmware.org%2Farchives%2Fsearch%3Fmlist%3Dtf-&data=05%7C 01%7Cvwadekar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTW FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 Mn0%3D%7C3000%7C%7C%7C&sdata=37mdLWDEDIRwAnsxo9c7bu2KwDq9oX%2B3Y4j %2BNiHzI%2Bc%3D&reserved=0 a%40lists.trustedfirmware.org&q=%5BTF-A%5D+ATF+LTS+version [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. trustedfirmware.org%2Fdocs%2FTF-A-LTS.pdf&data=05%7C01%7Cvwadekar% 40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d15727340c1b7db 39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbGZsb3d8eyJW IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% 7C%7C%7C&sdata=05dYPwDmH7OZMjMVb0AtN9eSJR8R4InaZ5ANo8qqqUA%3D& reserved=0 [3] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.trustedfirmware.org%2Farchives%2Flist%2Ftf-&data=05%7C01%7Cvwade kar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d15727340c1 b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbGZsb3d8 eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3 000%7C%7C%7C&sdata=jjsY%2B9EEZQiTqg5o9RLucSB5WzwsheWOOUIzXz5PRUQ%3 D&reserved=0 a@lists.trustedfirmware.org/thread/ZZ2DDIW2DS4B7QYHYOL7XE4IPUW7LUNT/ [4] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. trustedfirmware.org%2Fdocs%2FTForg_LTS_proposal.pdf&data=05%7C01%7 Cvwadekar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d1572 7340c1b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbG Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% 3D%7C3000%7C%7C%7C&sdata=q1Pk6ucv8RA7g3Yj59CMkrb2ZRY1abNHSiCSHJBOn hg%3D&reserved=0