Hi Soby et. al.,
I wanna kick off a little discussion about how TF-A intends to deal with in-tree platform ports as they get older and the interest in maintaining them drops off. Concretely, I noticed that the plat/nvidia/tegra platforms no longer build since the removal of the deprecated console API in https://review.trustedfirmware.org/842 last month. There has been a patch suggestion to fix it uploaded at https://review.trustedfirmware.org/1192 for two months, but it hasn't moved forward because it seems that Arm thinks it's on the platform maintainer (Nvidia) to finish up and test the patch, and they don't seem to be responding.
This creates a problem for downstream projects like coreboot and Chrome OS that use Trusted Firmware on Tegra chips and build-test them in their CI systems. My assumption when setting up the Trusted Firmware integration for them was that the Trusted Firmware CI would build test all in-tree platforms for every commit anyway, so we could always assume that all platforms build on the current master... but clearly, that assumption broke in this case. (I guess because you manually overrode the CI in https://review.trustedfirmware.org/842? Or does it not test all platforms anyway?) So now, coreboot is stuck on an old TF-A version and cannot move forward for any platform until we either kick out the Tegra SoCs or get the problem fixed in TF-A (which is a problem with the testing because I don't have a Tegra board on hand either).
How do you think we should solve issues like this? Is keeping platforms that don't build in the tree an intended state? Is there some deadline after which you intend to remove the platform completely? Or would it be better to just merge "best effort" commits like https://review.trustedfirmware.org/1192 that we think should do the right thing for the platform (and at least makes it build again), even if nobody is around to test it on real hardware?
To give some experience from the coreboot project, I think it's an unfortunate truth that SoC vendors just tend to lose interest in maintaining hardware once it's more than 2-3 years old. At that point the open-source community has to jump in to continue maintenance, and they can only do it on a best effort basis. It's not possible to always find someone with the right hardware and time to test it for all these old platforms whenever you're trying to do some large, project wide API change, so eventually you'll just have to accept patches that haven't been tested for them. Most of the times (if reviewers pay attention) it works well, sometimes they break. If they do, eventually someone will notice and then they'll have to bisect and fix it. I believe Linux is essentially doing the same thing for lesser-used hardware. It's either that, or you have to constantly kick out old platforms after a few years. (From the coreboot point of view, kicking the Tegra platforms out of TF-A would mean we're forced to remove them from coreboot as well, which would be unfortunate.)
Let me know what you think! Julius
Thanks for the email, Julius. To be clear, we very well intend to be part of the TF-A project. Having said that, I was not aware of the two commits you mentioned in the email and di not know that Tegra builds are broken in the master branch.
I will try to provide a fix for the build break ASAP.
-----Original Message----- From: Julius Werner jwerner@google.com Sent: Friday, August 2, 2019 2:43 PM To: Soby Mathew Soby.Mathew@arm.com; tf-a@lists.trustedfirmware.org Cc: Varun Wadekar vwadekar@nvidia.com; ambroise.vincent@arm.com Subject: On supporting unmaintained platforms and downstream projects
Hi Soby et. al.,
I wanna kick off a little discussion about how TF-A intends to deal with in-tree platform ports as they get older and the interest in maintaining them drops off. Concretely, I noticed that the plat/nvidia/tegra platforms no longer build since the removal of the deprecated console API in https://review.trustedfirmware.org/842 last month. There has been a patch suggestion to fix it uploaded at https://review.trustedfirmware.org/1192 for two months, but it hasn't moved forward because it seems that Arm thinks it's on the platform maintainer (Nvidia) to finish up and test the patch, and they don't seem to be responding.
This creates a problem for downstream projects like coreboot and Chrome OS that use Trusted Firmware on Tegra chips and build-test them in their CI systems. My assumption when setting up the Trusted Firmware integration for them was that the Trusted Firmware CI would build test all in-tree platforms for every commit anyway, so we could always assume that all platforms build on the current master... but clearly, that assumption broke in this case. (I guess because you manually overrode the CI in https://review.trustedfirmware.org/842? Or does it not test all platforms anyway?) So now, coreboot is stuck on an old TF-A version and cannot move forward for any platform until we either kick out the Tegra SoCs or get the problem fixed in TF-A (which is a problem with the testing because I don't have a Tegra board on hand either).
How do you think we should solve issues like this? Is keeping platforms that don't build in the tree an intended state? Is there some deadline after which you intend to remove the platform completely? Or would it be better to just merge "best effort" commits like https://review.trustedfirmware.org/1192 that we think should do the right thing for the platform (and at least makes it build again), even if nobody is around to test it on real hardware?
To give some experience from the coreboot project, I think it's an unfortunate truth that SoC vendors just tend to lose interest in maintaining hardware once it's more than 2-3 years old. At that point the open-source community has to jump in to continue maintenance, and they can only do it on a best effort basis. It's not possible to always find someone with the right hardware and time to test it for all these old platforms whenever you're trying to do some large, project wide API change, so eventually you'll just have to accept patches that haven't been tested for them. Most of the times (if reviewers pay attention) it works well, sometimes they break. If they do, eventually someone will notice and then they'll have to bisect and fix it. I believe Linux is essentially doing the same thing for lesser-used hardware. It's either that, or you have to constantly kick out old platforms after a few years. (From the coreboot point of view, kicking the Tegra platforms out of TF-A would mean we're forced to remove them from coreboot as well, which would be unfortunate.)
Let me know what you think! Julius
----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
Thanks for the email, Julius. To be clear, we very well intend to be part of the TF-A project. Having said that, I was not aware of the two commits you mentioned in the email and di not know that Tegra builds are broken in the master branch.
Thanks. You were CCed on the patch so I assumed you would've seen it. If not, maybe your email address isn't set correctly in your Gerrit account or something? I've already pushed an update to https://review.trustedfirmware.org/1192 which I think should fix the issue for Tegra, but I need someone to test it.
Nevertheless, I think it's a good idea to answer these questions in a general case (e.g. whether we can make sure that we won't break the build on master even if there are temporary issues with certain platforms), because it's probably going to become relevant again sooner or later even if the Tegra issue gets fixed now.
I checked the Gerrit project settings and saw that my email is not set. I will try to update my profile.
Thanks for pointing that out.
-----Original Message----- From: Julius Werner jwerner@google.com Sent: Friday, August 2, 2019 5:10 PM To: Varun Wadekar vwadekar@nvidia.com Cc: Soby Mathew Soby.Mathew@arm.com; tf-a@lists.trustedfirmware.org; ambroise.vincent@arm.com Subject: Re: On supporting unmaintained platforms and downstream projects
Thanks for the email, Julius. To be clear, we very well intend to be part of the TF-A project. Having said that, I was not aware of the two commits you mentioned in the email and di not know that Tegra builds are broken in the master branch.
Thanks. You were CCed on the patch so I assumed you would've seen it. If not, maybe your email address isn't set correctly in your Gerrit account or something? I've already pushed an update to https://review.trustedfirmware.org/1192 which I think should fix the issue for Tegra, but I need someone to test it.
Nevertheless, I think it's a good idea to answer these questions in a general case (e.g. whether we can make sure that we won't break the build on master even if there are temporary issues with certain platforms), because it's probably going to become relevant again sooner or later even if the Tegra issue gets fixed now.
----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
On 02/08/2019 22:43, Julius Werner wrote:
Hi Soby et. al.,
I wanna kick off a little discussion about how TF-A intends to deal with in-tree platform ports as they get older and the interest in maintaining them drops off. Concretely, I noticed that the plat/nvidia/tegra platforms no longer build since the removal of the deprecated console API in https://review.trustedfirmware.org/842 last month. There has been a patch suggestion to fix it uploaded at https://review.trustedfirmware.org/1192 for two months, but it hasn't moved forward because it seems that Arm thinks it's on the platform maintainer (Nvidia) to finish up and test the patch, and they don't seem to be responding.
This creates a problem for downstream projects like coreboot and Chrome OS that use Trusted Firmware on Tegra chips and build-test them in their CI systems. My assumption when setting up the Trusted Firmware integration for them was that the Trusted Firmware CI would build test all in-tree platforms for every commit anyway, so we could always assume that all platforms build on the current master... but clearly, that assumption broke in this case. (I guess because you manually overrode the CI in https://review.trustedfirmware.org/842? Or does it not test all platforms anyway?) So now, coreboot is stuck on an old TF-A version and cannot move forward for any platform until we either kick out the Tegra SoCs or get the problem fixed in TF-A (which is a problem with the testing because I don't have a Tegra board on hand either).
Hi Julius,
Thanks for raising this; this is indeed a discussion we need to have. As you figured, we had to manually disable Tegra from internal CI builds because it is not keeping up with changes upstream.
Just to be clear, in addition to the gerrit notifications, we had given enough notification regarding the removal of legacy console. See [1][2][3] and also the deprecation timeline [4].
[1] https://github.com/ARM-software/tf-issues/issues/692 [2] https://lists.trustedfirmware.org/pipermail/tf-a/2019-May/000021.html [3] https://lists.trustedfirmware.org/pipermail/tf-a/2019-May/000022.html [4] https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/about/docs/proce...
How do you think we should solve issues like this? Is keeping platforms that don't build in the tree an intended state? Is there some deadline after which you intend to remove the platform completely? Or would it be better to just merge "best effort" commits like https://review.trustedfirmware.org/1192 that we think should do the right thing for the platform (and at least makes it build again), even if nobody is around to test it on real hardware?
To give some experience from the coreboot project, I think it's an unfortunate truth that SoC vendors just tend to lose interest in maintaining hardware once it's more than 2-3 years old. At that point the open-source community has to jump in to continue maintenance, and they can only do it on a best effort basis. It's not possible to always find someone with the right hardware and time to test it for all these old platforms whenever you're trying to do some large, project wide API change, so eventually you'll just have to accept patches that haven't been tested for them. Most of the times (if reviewers pay attention) it works well, sometimes they break. If they do, eventually someone will notice and then they'll have to bisect and fix it. I believe Linux is essentially doing the same thing for lesser-used hardware. It's either that, or you have to constantly kick out old platforms after a few years. (From the coreboot point of view, kicking the Tegra platforms out of TF-A would mean we're forced to remove them from coreboot as well, which would be unfortunate.)
Let me know what you think! Julius
Hmm, if we merge a non-trivial patch and ensure the build works, then we do not know whether it runs correctly, whether there are any runtime effects that would affect stability/robustness of the platform that even might have security implications. Hence, in my view, it is better to have a broken build for the platform, rather than have runtime problems.
The project could form a policy that if a platform remains broken for more than 2 releases (1 year by current release intervals), then it will get removed from the tree after giving enough notifications.
We already have a mediatek platform in the same state (mt6795) and several attempts to contact the maintainer has failed.
But I agree we need to have a policy for this.
Best Regards Soby Mathew
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
tf-a@lists.trustedfirmware.org
-
Julius Werner -
Soby Mathew -
Varun Wadekar