Sandrine,
To clarify on functionality vs. support. I listed out a support life cycle consisting of the following states: Fully Supported Orphan Out of Date Deprecated These states are intended to have nothing to do with functionality, but only the support offered for the functionality that currently exists for a platform.
I think I may have confused things when I listed "Functional Support" as the heading to represent "Functionality." I'm proposing that the supported "Functionality" should be documented in a standard way (within a project) for every platform.
I do agree this could be burdensome to keep up with. But that is why I suggested that the project's feature list be versioned. The platform's supported feature list document would reference the version of the project feature list used. Platform maintainers then don' t have to continuously update the document. But it will be clear how long it has been since they did update and thus what information may be missing. Versioning the feature list document is also why I mentioned that the project version number may want to adopt a version number scheme where feature changes are represented by a certain part of the version number. For example Semantic Versioning 2.0.0: https://semver.org/. Hope that clarifies the intent? For implementation of this I'm imagining each project could create a supported_feature_list.rst file and each platform would copy that file into their platform doc folder and fill it in. I'm not saying that approach would be required at tf.org level, just sharing to further illustrate.
That said, perhaps the implementation details for a project would not warrant such a document per platform? My primary concern around this is misuse/misconfiguration. If a platform doesn't support a feature or configuration it may not be obvious to a user unless an error is generated at build or run-time.
My secondary concern is being able to consistently track tickets/bugs with features. Thus, I'm recommending that the features on that list be used with the ticket/issue system by feature name. This would allow users to find all bugs for Feature X on Platform Y in Project Z. Related to that, when I mentioned "tags." I wasn't thinking of Git tags, but "labels" in the ticket/issue tracking system(s). Different systems work differently for labeling/categorizing issues, but the goal is to provide a consistent way (per project) to find issues related to a feature on a platform.
If requiring a feature list it is too much at the tf.org level then I'll be satisfied to push for that kind of documentation in the projects or platforms I'm involved in if/as appropriate.
Regarding the other conversational tidbits:
Thanks for pointing out that the original proposal does say "builds all configurations supported by this platform" under "Fully Supported." I can see the intention here now. Substituting "features" for "configurations" would broaden the meaning a bit.
You said: "I am starting to think that we need a list of items to be defined per project." Yes, this sounds like a great idea.
My original mention of wanting a "stronger standard put forth for platform documentation" was a response to seeing "Limited Support" in the original proposal allowing documentation to fall out of date.
Hope that clarifies some of my thoughts. If not, I'm happy to continue the discussion. Thanks again for taking feedback!
Erik Shreve, PSEM Software Security Engineer & Architect (CMCU Platform Development)
-----Original Message----- From: Sandrine Bailleux [mailto:sandrine.bailleux@arm.com] Sent: Wednesday, April 01, 2020 4:18 AM To: Shreve, Erik; tf-a; tf-m@lists.trustedfirmware.org; tsc@lists.trustedfirmware.org; op-tee@linaro.org Cc: nd@arm.com Subject: Re: [EXTERNAL] [TF-M] Project Maintenance Proposal for tf.org Projects
Hello Erik,
Thanks for the feedback.
On 3/26/20 3:37 PM, Shreve, Erik wrote:
Sandrine,
Really glad to see this being pulled together. A couple of areas of feedback around the Platform Support Life Cycle.
As previously mentioned there are two orthogonal concerns captured in the current life cycle: Support and Functionality. I'd like to see these split out.
Yes, you are the second person to mention that and I agree with you both. Unless someone disagrees, I intend to update the proposal and separate these 2 concepts in the next version of the document.
For functionality, chip vendors may not have a business case for supporting all features on a given platform but they may provide full support for the features they have chosen to include. A simple example would be supporting PSA FF Isolation Level 1 only due to lack of HW isolation support needed to achieve Isolation Level 2 or greater.
I completely agree. It would not make sense to support all features on all platforms just for the sake of completeness. Each platform ought to implement what is relevant in its case.
That's what the current proposal tried to convey: a fully supported platform must "build all configurations *supported by this platform*" and "All *supported* configurations are tested in the CI". The key word here is supported and that would be defined by the platform itself. But I can see that maybe this wasn't clear enough. Your proposal below makes that a lot clearer.
Also, I'd like to see a stronger standard put forth for platform documentation. If a platform is "supported," I believe the documentation should be complete and accurate. A lack of complete and clear documentation leaves open a wide door for misuse/misconfiguration which could result in a vulnerable system.
Fair point.
But is it something we should include in this proposal or should we push it to a separate document setting expectations for the project's documentation, which the current general proposal could refer to (as in, "the platform should provide quality documentation up to the project's criteria defined in document XXX')?
This is definitely an important topic but I am wary of keeping the tf.org proposal concise and focused at this point. I am worried that if we put too much stuff in it discussions will diverge too much and we might never reach an agreement.
The same applies to testing standards for example, we could detail that in the proposal or simply leave it to projects to define it separately.
Here is a more concrete proposal:
Functional Support: Each project shall provide a standard feature or functionality list. Each platform shall include in its documentation a copy of this list with the supported functionality marked as supported. The platform documentation may reference a ticket if support is planned but not yet present. The platform documentation shall explicitly state if a feature or function has no plans for support.
Regarding the last item, this would require all platform maintainers to update their documentation every time a new feature is added to the project's global list of features. This seems too much of a constraint and unnecessary maintenance burden to me.
I think a better, more lightweight alternative might be to let platform maintainers list what's supported and if some feature is not listed, it implies that it is not supported. This does not prevent platform maintainers from indicating their future plans of supporting a feature if they want to.
The feature/functionality list shall be versioned, with the version tied to the release version(s) of the project. In this way, it will be clear if a platform was last officially updated for version X but the project is currently at version Y > X.
I can see that Joakim Bech proposed something similar, with more details about how this was implemented for OP-TEE.
Note: projects will need to adopt (if they have not already) a version scheme that distinguishes between feature updates and bug fixes.
Sorry I didn't get this, could you please elaborate?
Each project and platform shall use tags or similar functionality on tickets to associate tickets to features/functionality and platforms. If the names of tags can't match the name of the feature or platform exactly then a mapping shall be provided in the appropriate document(s).
If there's no appropriate tag in some cases, I guess we could always use a git SHA1 of a specific commit.
Life Cycle State
Fully Supported There is (at least) one active code owner for this platform. All supported features build and either all tests pass or failures are associated with tracked known issues. Other (not associated to a test) Known Issues are tracked Documentation is up to date
Note: Projects should document standards on how "active" code ownership is measured and further document standards on how code owners are warned about impending life cycle state changes.
Yes, good point, that is currently undefined in the proposal but I agree that it needs defining per project. I will add an item in the last section of the document.
I am starting to think that we need a list of items to be defined per project. This list would complement the general tf.org proposal. Things like code owners/maintainers activity, code review timelines, and so on.
Orphan There is no active code owner All supported features build and either all tests pass or failures are associated with tracked known issues. Other (not associated to a test) Known Issues may not have been maintained (as there is no active code owner) Documentation status is unclear since there is no active code owner. There has been no change to the feature/functionality list in the project since the platform was last "Fully Supported"
I am confused, you said earlier that you would like to see the concepts of support and functionality split out, but here you're listing 'orphan' as one of the possible states... Did I miss your point?
Out of date Same as orphan, but either: there have been changes to the feature/functionality list, or there are failing tests without tracked tickets, or there are known documentation issues.
Deprecated Same as Out of Date, but the build is broken. Platform may be removed from the project codebase in the future.
Erik Shreve, PSEM Software Security Engineer & Architect (CMCU Platform Development)