OK, this seems to be the crux of the problem. Is it possible for us say that users are free to register new TLs, while at the same time recommending the use of existing formats for complex structures (e.g. FDT, HOB)?
I'm not really sure what qualifies as a "complex structure" in this discussion? I think each individual TE should deal with a single concern, and separate concerns should be implemented via separate TEs (IIRC I tried to propose language in that direction in my PR)... so I agree that TEs shouldn't be "complex" (if this is what you mean by that), but I think the solution to that is just to divide the complexity down into a larger number of more simple TEs. I also agree that usually each TE should be representable by a simple C structure like I think Simon mentioned somewhere (although I think that should be a recommendation instead of an absolute requirement in case projects have long-established data structures that they want to continue using... the FDT and HOB list TEs aren't representable by a C structure either, after all).
On the other hand, if this is just supposed to be an early boot flow extension to FDTs, then that's very different from what I understood the goal to be and then I think the text of the spec should be honest about that front and center. In that case, I wouldn't expect much adoption beyond the ecosystem that's already deeply tied to FDT to begin with, though, and that would be far from "universal".
That is another valid usage model. Some ecosystems are deeply wedded to FDT or HOB/ACPI (there may be others) and this spec is not going to change that. I don't think we're going to get something universal in the way you're hoping. But we might be able to at least enable better interoperability/reusability between fw components across different ecosystems.
Sure, I didn't mean to say this should be disallowed, but the question is whether that is the only allowed use and the FDT is required to pass certain data, or whether adopters are free to choose how they represent their data and the FDT tag is just one of many. Like you said there really needs to be maintainer consensus about this that must be written down somewhere.
I don't believe there's going to be a lot of sharing between projects either to be honest, but ultimately that's what this whole thing was started for, right? I think in practice the most likely opportunities for sharing are going to be from small pieces that all projects need, like a TE defining which serial console to print debug output on. (That's another good reason to keep TEs simple, single-concern, and directly encoding data instead of wrapping another structure. In practice I assume that the projects that mostly rely on the wrapped HOB list or FDT will duplicate a few pieces of data into separate TEs to be able to share those with components that can't parse the larger structure.)
A tag that is intended to be used for information passed between different firmware projects. Such as from TF-A to EDK2 and u-boot.
A tag for _internal_ use within a firmware project. Here a firmware project should be free to do whatever they want, but they still will likely want to use tag IDs that will not conflict with other uses. I don't see the value of cluttering the firmware handoff spec with layouts internal to specific firmware projects.
This solves the accidental overlap concern but not the organically emerging standard concern. I guess we could say that a tag from the "vendor" range can be promoted to "standard" at a later point by adding it to the global spec (with its existing ID in the vendor range). This does hurt discoverability though, i.e. it's harder for someone who is trying to implement a new TE to realize that the problem has already been solved by another project in a way that would also work out well for them.
I still think it's important that tag layouts must be immutable and cannot change once allocated if they come out of that range, that's the basis for any interoperability and backwards-compatibility. And the best way to ensure that is to have them all listed in a global location. Otherwise, even if you write down somewhere that layouts mustn't change, tags shouldn't be reused after they get deprecated, etc., the fact that nobody would notice if you do will encourage people to silently do it anyway.
I agree that we don't want to have all of them clutter a single .rst file once there are hundreds or thousands, but there are ways to organize that better which we can decide on once we get to that point.