Hi,
Let's continue to discuss this topic.
I agree with this, the CI is just a tool, we should not only rely on it.
Thanks Gyorgy to point out the quality policy, we need to think about them when we discuss the version control.
In the current status, there is only one master branch for most of the development work in TF-M. Of course, we can add a warning to say that the master is constantly changing, there may be some problems in building or test running, and let the user use the
release tag. But it is not convenience and impossible if they are upstreaming patches. And even to TF-M developers, we had met several times of the master branch is broken.
There is another way maybe help to solve this problem, we can define a merge policy like this: The patches cannot be merged to master branch if the patch is not based on the TOP HEAD. But this is very difficult to follow because there may be several maintainers
could and need to merged patches at the same time. They need to align with each other to confirm their patches are on the TOP. But now, we often meet this case, while one patch rebases to TOP and waiting for the CI result, another patch is merged into master.
The patch needs to rebase again.
I do not want to give more examples here.
Of course, it is a big change to involve a new branch, there must be many documents that need to be updated, and some policies need to be changed. And even this needs someone to maintain the alignment with 2 branches. But I think it is more useful and helpful
for all users.
Thanks,
Edison
-----Original Message-----
From: TF-M <tf-m-bounces@lists.trustedfirmware.org> On Behalf Of Gyorgy Szing via TF-M
Sent: Friday, December 13, 2019 10:01 PM
To: Minos Galanakis <Minos.Galanakis@arm.com>; Soby Mathew <Soby.Mathew@arm.com>;
tf-m@lists.trustedfirmware.org
Cc: nd <nd@arm.com>
Subject: Re: [TF-M] Create another branch for feature development
Hi,
Hi,
I agree, the CI shall not dictate how we use the version control system. It shall adapt.
Regarding your suggestions, I think the main problem is we are mixing stuff, this time quality with version control. Before we make decisions we shall understand where we are.
The current quality policy is that we only make releases for communication purposes. To give a clean interface for tf-m users and to allow planning their work. Releases allow them to execute their tf-m integration process less frequently. Only for each release
or specific releases and not for each commit. The current quality policy identifies a single quality level only, and says any patch we publish is "golden quality", it matches the highest quality standard we can achieve (with sane constraints). Also to make
our life easy we decided to use the master branch to hold these patches.
At the same time we use the master branch for development. Any change we make is made against master. This means each pull request and thus each review targets master. For review purposes the best is to have a chain of small modifications, otherwise the review
content becomes too large to follow.
The TF-A "branching strategy" tries to address this issue by introducing an integration branch used for development. This allows master to be more release specific.
I suggest to take the following approach (details to be discussed):
- introduce more quality levels i.e.:
- none: content of a topic branch, or content pushed to review.
- bronze: content passed code review and patch specific testing.
- silver: content passed a more though daily testing.
- gold: a release. A pack of source-code, feature state document (release notes), reviewed documentation (user manual, reference manual), test evidence, documentation of test efforts to allow repeatability. The version control system can be used to store
content, and to provide identification info (i.e. tagging), but most likely the release will need other kind of storage to be used (i.e. documentation).
- platina: reaching extra quality level trough passing PSA or some FUSA qualification. Or we may simply use extra release for this.
Naming the quality levels allows us to have a cleaner definition of what can be expected at a specific level (set of quality measures, i.e. static analysis, code review, test configuration). It would also allow us cleaner communication and to find how we use
the version system for quality purposes.
I also expect this discussion to help defining how the version system is used for development purposes.
The current state works ok, but is a sort of naturally grown. We might have reached the point when more pragmatic approach may be needed.
/George
-----Original Message-----
From: TF-M <tf-m-bounces@lists.trustedfirmware.org> On Behalf Of Minos Galanakis via TF-M
Sent: 13 December 2019 12:23
To: Edison Ai (Arm Technology China) via TF-M <tf-m@lists.trustedfirmware.org>; Soby Mathew <Soby.Mathew@arm.com>
Cc: nd <nd@arm.com>
Subject: Re: [TF-M] Create another branch for feature development
Hi all.
My personal comments on this.
I would like to point out that the CI is a tool, not the core project. I do not believe we should be changing our development strategy based on what the tool is doing. We should instead adjust the tool to fit our requirements.
* No patches should be/ are merged to master when CI fails. If master breaks it should most commonly be because of something we are not testing for. Using an integration branch would not change that.
* As a developer I find it more convoluted to work with projects who use different integration strategies. The most common assumption in open source projects is that you have a master branch which is the bleeding edge, but can contain untested bugs, and
the release immutable git tags for versioning. Using branch merges as versioning is a design for the pull request model which is not quite compatible with Gerrit.
* Most of the CI downtime has nothing to do with the merge strategy, they are more of a chicken and the egg philosophical problem. If your patch or branch introduced a change which affects the tests outputs, how will you test it if the CI expects the old
output? An integration branch would not solve the merge freeze periods, would just affect a different branch from master.
* I believe feature branches are quite useful, since changes to master do not disrupt the development flow of a big change, and even though they will require some maintenance to re-sync before the final patch , it will be handled by an engineer who knows
the feature, and full understands the regression vectors.
If I were to suggest some changes for stability purposes, I would start smaller:
* Update documentation to instruct users to check out from release tags, warning then that master is constantly changing.
* Adjust the CI to detect an Allow-CI flag from every branch. That way developers can test any patch from any feature branch. The logic for that is already present in the code, but requires Gerrit to be configured accordingly.
* Add an undo process. This would be the only case for an integration branch. All patches are merged to a temporary branch, after confirming they have passed testing individually. On the once per day nightly test, the group of different patches, will be
tested against an extensive job, in models and hardware, and only if successful it will fast forward master to that state.
Regards
Minos
________________________________
From: TF-M <tf-m-bounces@lists.trustedfirmware.org> on behalf of Edison Ai (Arm Technology China) via TF-M <tf-m@lists.trustedfirmware.org>
Sent: 13 December 2019 08:55
To: Soby Mathew <Soby.Mathew@arm.com>; 'tf-m@lists.trustedfirmware.org' <tf-m@lists.trustedfirmware.org>
Cc: nd <nd@arm.com>
Subject: Re: [TF-M] Create another branch for feature development
Hi Soby,
Thanks for your detail description.
> Integration is a temporary merge branch to merge several patches and run the CI against. Usually once CI passes, the master will be fast forwarded to integration within a day.
> This helps us to test integration of patches and detect any failure before master is updated. This means the master will pass CI at any given merge point.
I think it's a good method like this so that we can double confirm the "master" branch is stable.
And this also can fix one case even the CI can work normally: one patch is ready to merge, and it is not based on the latest HEAD, but there is no conflict. We can merge the patch directly and let gerrit do rebase by itself. But we cannot confirm the CI test
can pass.
Any comment for this from others?
For multiple feature branches, I think we can stop to discuss about it now until we have some strong demands for it. It is indeed a big change for us now.
Thanks,
Edison
-----Original Message-----
From: Soby Mathew <Soby.Mathew@arm.com>
Sent: Friday, December 13, 2019 5:14 AM
To: Edison Ai (Arm Technology China) <Edison.Ai@arm.com>; 'tf-m@lists.trustedfirmware.org' <tf-m@lists.trustedfirmware.org>
Cc: nd <nd@arm.com>
Subject: Re: [TF-M] Create another branch for feature development
On 11/12/2019 09:05, Edison Ai (Arm Technology China) via TF-M wrote:
> Hi Gyorgy,
>
> Thanks to point it out. I agree with you that it will be better if we can align these two projects in this. I had a quick check the branches from TF-A:
https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/.
> There are three branches in TF-A:
> - "integration" branch, should be used for new features.
> - "master" branch, which is behind of "integration" branch. But I am nor sure what is the strategy to update it.
Hi Edison,
Integration is a temporary merge branch to merge several patches and run the CI against. Usually once CI passes, the master will be fast forwarded to integration within a day.
This helps us to test integration of patches and detect any failure before master is updated. This means the master will pass CI at any given merge point.
> - "topics/epic_beta0_spmd", I thinks it should like a feature branch for big feature.
> @Soby Mathew Could you help to share more information about it? Thanks very much.
We usually do not have feature branches in TF-A. The topics/epic_beta0_spmd is a prototyping branch where we wanted to share code with collaborators outside TF-A. The patches on this branch are not visible in gerrit review and no patches in gerrit review will
be merged to this branch. Once the prototyping is complete, then patches on this branch will be reworked and pushed to gerrit review and finally get merged to integration and this branch will be deleted.
Our experience have been, long running development branches are generally a maintenance overhead. Merging these development branches before a release may also be risky as some of the changes may have unknown interactions and may become difficult to resolve.
The "topic" in gerrit review is effectively a branch. For example, this
review:
https://review.trustedfirmware.org/#/q/topic:od/debugfs+(status:open+OR+status:merged)
is a set of patches on topic "od/debugfs" and can be treated as development branch. This branch is alive as long as patches are not merged.
We need to understand the motivations for the change. Broken CI is an argument but development branches will only exacerbate that problem since we don't know the stability of each of those branches. Also merge conflict will not reduce due to development branches.
Its just delaying the merge conflict to a later point.
There may be other reasons, but generally it is better to merge sensible patches (+2ed) within a feature even before the feature is complete as it will reduce merge conflicts (we have to ensure testing/build coverage for the patch). These are my thoughts on
this.
Best Regards
Soby Mathew
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
--
TF-M mailing list
TF-M@lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m