Re: [TF-TSC] Finally, a TF-A LTS Proposal!
by Varun Wadekar
Hello,
I have uploaded the proposal doc as a pdf to the LTS Phabricator page [1]. Please review and provide feedback.
Thanks.
[1] https://developer.trustedfirmware.org/w/tf_a/lts_proposal/
-----Original Message-----
From: Varun Wadekar
Sent: Tuesday, 6 September 2022 1:24 PM
To: Dan Handley <Dan.Handley(a)arm.com>; Okash Khawaja <okash(a)google.com>; tf-a(a)lists.trustedfirmware.org
Cc: tsc(a)lists.trustedfirmware.org
Subject: RE: [TF-TSC] Finally, a TF-A LTS Proposal!
Hi,
Thanks for the feedback. My comments below.
> There's no mention of normal (i.e. non-security) bugs in generic code. Presumably these don't meet the criteria (?), but it's better to be explicit.
[VW] Makes sense. The intention is to focus on security issues, but I agree that we should look at such bugs on a case-by-case basis. We will try to include this in the guidance.
> I know with the Linux LTS branches that patches are tagged for backporting in the mainline commit messages, and scripts automatically backport to the LTS branches. When patches fail to cleanly backport, it's up to the LTS community to address as it's not scalable for the LTS maintainer(s) to fix all problems. We could consider copying this model (even the scripts) eventually.
[VW] That is a good idea. I agree we should try to replicate mechanisms that already work.
> I guess you mean there needs to be more downstream testing of the mainline release before the first LTS release is made. That's fine but this should be made clear and there should be a fixed window so that downstream testing doesn't gate the LTS release.
[VW] The intention was to allow platforms more testing time. We have tried to provide a deadline but will try to make it clear.
> I think it's reasonable to make all TF-A contributors responsible for identifying such patches at the time they created, with the LTS maintainer acting as backup when things are missed.
[VW] This is what I had in mind at the start and want to aim for. But, want to catch scenarios where contributors miss this. So, we want a reviewer list that never misses such a patch and then educate the contributors to get to the end goal.
> "After reviews are complete, merge the patch and bump the minor version, if required". This suggest no downstream testing is required for minor LTS versions - is that right? I would have thought a small window is required. Or can we assume that further patches/releases are made if there are problems?
[VW] The intent is to complete the testing _before_ the patch is merged to the branch. This testing again has a deadline to avoid slowing down the release process.
> "What happens when a bug fix applies just to a LTS branch and not to the master branch?". If a bug is only reproducible on the LTS branch, then it seems to me this is a reasonable exception to the "only backport from mainline" rule.
[VW] Agreed.
> "When testing a backported patch, what if one of the partners needs more time while the patch fix is time-critical and hence slowing other partners?". I would say there should be a fixed window. If a partner subsequently finds a problem, another minor release can be made.
[VW] Agreed.
> I suspect a new TF-A LTS mailing list will be required for this
[VW] Probably. Would like to address this in the Tech forum.
> This is going to take more discussion, but my personal view is the current set of maintainers/contributors does not closely intersect with those interested in consuming an LTS, so it's wouldn't be fair for the current maintainers/contributors to take the whole burden of LTS maintainership.
[VW] I understand. In the absence of a formal group, the easiest was forward was to request the current maintainers. We should discuss more in the Tech forum.
-Varun
-----Original Message-----
From: Dan Handley <Dan.Handley(a)arm.com>
Sent: Thursday, 11 August 2022 12:12 PM
To: Okash Khawaja <okash(a)google.com>; tf-a(a)lists.trustedfirmware.org
Cc: tsc(a)lists.trustedfirmware.org; Varun Wadekar <vwadekar(a)nvidia.com>
Subject: RE: [TF-TSC] Finally, a TF-A LTS Proposal!
External email: Use caution opening links or attachments
Hi Okash/Varun/all
Thanks for this. Some comments from me:
* Criteria: I agree with Joakim that in some cases, a feature may need to be backported as a dependency, although I think this can be mainly decided by the LTS maintainer(s) on a case by case basis. TSC should only be needed where there's controversy/disagreement. There's no mention of normal (i.e. non-security) bugs in generic code. Presumably these don't meet the criteria (?), but it's better to be explicit.
* Frequency: Yearly in November makes sense but if they live for 5/7 years that obviously means 5/7 concurrent branches to maintain, which will make automation even more important.
* Regarding automation, clearly all the CI needs to be automated from the start. It may be possible to manually backport to start with but even this will need to be mostly automated if there are many concurrent branches. I know with the Linux LTS branches that patches are tagged for backporting in the mainline commit messages, and scripts automatically backport to the LTS branches. When patches fail to cleanly backport, it's up to the LTS community to address as it's not scalable for the LTS maintainer(s) to fix all problems. We could consider copying this model (even the scripts) eventually. You mention tagging patches on mainline instead, which could work but I guess could result in tag proliferation.
* Testing criteria: +1 on Joakim's comment about downstream testing not gating LTS patch merging. I'd also say that platforms can only be tested in the LTS CI if they are also tested in the mainline CI.
* Test-and-debug period: "it is expected that more testing and debugging will be done on the November release of TF-A". The testing criteria in the previous section suggest LTS tests are a subset of mainline testing, so in theory the LTS release could be made at the same time as main release. I guess you mean there needs to be more downstream testing of the mainline release before the first LTS release is made. That's fine but this should be made clear and there should be a fixed window so that downstream testing doesn't gate the LTS release.
* "[LTS Maintainer to] Monitor the main branch to identify candidate patches for the LTS branches". This might work to start with but I don't think is very scalable. I think it's reasonable to make all TF-A contributors responsible for identifying such patches at the time they created, with the LTS maintainer acting as backup when things are missed.
* "After reviews are complete, merge the patch and bump the minor version, if required". This suggest no downstream testing is required for minor LTS versions - is that right? I would have thought a small window is required. Or can we assume that further patches/releases are made if there are problems?
* "What happens when a bug fix applies just to a LTS branch and not to the master branch?". If a bug is only reproducible on the LTS branch, then it seems to me this is a reasonable exception to the "only backport from mainline" rule.
* "When testing a backported patch, what if one of the partners needs more time while the patch fix is time-critical and hence slowing other partners?". I would say there should be a fixed window. If a partner subsequently finds a problem, another minor release can be made.
* I suspect a new TF-A LTS mailing list will be required for this (TBD).
* Maintainership options: This is going to take more discussion, but my personal view is the current set of maintainers/contributors does not closely intersect with those interested in consuming an LTS, so it's wouldn't be fair for the current maintainers/contributors to take the whole burden of LTS maintainership.
Regards
Dan.
> -----Original Message-----
> From: Okash Khawaja via TSC <tsc(a)lists.trustedfirmware.org>
> Sent: 08 August 2022 18:49
> To: tf-a(a)lists.trustedfirmware.org
> Cc: tsc(a)lists.trustedfirmware.org; Varun Wadekar <vwadekar(a)nvidia.com>
> Subject: [TF-TSC] Finally, a TF-A LTS Proposal!
>
> TL;DR: Attached is the TF-A LTS proposal doc. Please review and share
> your thoughts.
>
> Hello everyone,
>
> Long term support for TF-A has a long history. As a community we have
> flirted with it for a while, and like any flirtation, it started with
> gossip -- in our case, over mailing list [1] -- more than two years
> ago. Then Varun Wadekar did a wonderful tech forum presentation [2]
> which demonstrated interest in the topic and raised interesting questions about LTS.
>
> After some time, the affair lost headline status, only to be revived
> again [3], earlier this year. This time however, there is a formal
> proposal (no pun intended). After the mailing list discussion and
> feedback from TF-A tech forum [4], we have put together a draft which
> attempts to give a concrete idea about what the TF-A LTS will look like.
>
> Please spare some time to read through it and share your feedback. The
> plan is to put LTS into action this November.
>
> Cheers!
> Okash
>
> [1]
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.trustedfirmware.org%2Farchives%2Fsearch%3Fmlist%3Dtf-&data=05%7C
> 01%7Cvwadekar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d
> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTW
> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> Mn0%3D%7C3000%7C%7C%7C&sdata=37mdLWDEDIRwAnsxo9c7bu2KwDq9oX%2B3Y4j
> %2BNiHzI%2Bc%3D&reserved=0
> a%40lists.trustedfirmware.org&q=%5BTF-A%5D+ATF+LTS+version
> [2]
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> trustedfirmware.org%2Fdocs%2FTF-A-LTS.pdf&data=05%7C01%7Cvwadekar%
> 40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d15727340c1b7db
> 39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbGZsb3d8eyJW
> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%
> 7C%7C%7C&sdata=05dYPwDmH7OZMjMVb0AtN9eSJR8R4InaZ5ANo8qqqUA%3D&
> reserved=0 [3]
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.trustedfirmware.org%2Farchives%2Flist%2Ftf-&data=05%7C01%7Cvwade
> kar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d15727340c1
> b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
> 000%7C%7C%7C&sdata=jjsY%2B9EEZQiTqg5o9RLucSB5WzwsheWOOUIzXz5PRUQ%3
> D&reserved=0
> a(a)lists.trustedfirmware.org/thread/ZZ2DDIW2DS4B7QYHYOL7XE4IPUW7LUNT/
> [4]
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> trustedfirmware.org%2Fdocs%2FTForg_LTS_proposal.pdf&data=05%7C01%7
> Cvwadekar%40nvidia.com%7Cdb167e872d95420c503408da7b8a5417%7C43083d1572
> 7340c1b7db39efd9ccc17a%7C0%7C0%7C637958131793859497%7CUnknown%7CTWFpbG
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C3000%7C%7C%7C&sdata=q1Pk6ucv8RA7g3Yj59CMkrb2ZRY1abNHSiCSHJBOn
> hg%3D&reserved=0
2 years, 9 months