Hello, There's an effort inside of Google to make builds more hermetic across the board. One of the challenges we ran into was supporting the transitive dependencies of generate_test_list.pl, namely XML::LibXML. Since this isn't in the perl standard library, we have to add what amounts to an entire sysroot with a bunch of dependencies just to support this one library. Not impossible (and to be clear, this is the route that we went with), but not ideal either.
Another approach that we explored was rewriting generate_test_list.pl in python. XML parsing is in python's standard library and many of our engineers are more familiar with it, so we thought this would be a reasonable choice. We wrote the script, and it seems to work (with some caveats that I'll discuss below). However, we don't want to diverge from upstream, hence why I'm here :).
The script is approximately the same length as the original script and supports all of the same features and checking, with one catch. The standard library's XML parsing module doesn't support entity expansion, which is a feature used in many places within TFTF (see, for example, tests-standard.xml). This is done for security reasons and I haven't found a way around it without resorting to xml parsers outside of the standard library (which kinda defeats the purpose). To resolve this, we use xincludes instead, meaning that tests-standard.xml (for example) would have to be rewritten to use the new syntax.
For example, tests-standard.xml would look like the following: <document> <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="tests-tftf-validation.xml"/> ... rest of includes here... </document>
Any thoughts on this? If you're okay with this approach we can put our CL up for review.
We also experimented with doing some hand-parsing to support entity expansion and I suspect that there's *some* way to convince entity expansion to occur within the standard library, but wanted to take your temperature on the above changes before pursuing those further.
Thanks, Jim