Publish the history of the reports somewhere

[mithro]

In #770 @wsnyder pointed out that it would be useful to have old reports published somewhere.

It appears the history is stored in the gh-pages at https://github.com/SymbiFlow/sv-tests/commits/gh-pages

We could do this by just always committing to a new directory and updating an "index" symlink, git will automatically dedup identical files in the repo. See the https://github.com/timvideos/HDMI2USB-firmware-prebuilt/tree/master/archive/master repository for example.

[mithro]

@hzeller Any thoughts on what we should do here?

[hzeller]

I'd actually like to run all the tools in the different versions against the current set of tests. The tests were improved over time, have had fixes where there were plain wrong or tested the wrong thing, and also we extended the set of test-cases with new cores etc.

So to have a nice apples-to-apples comparison, best would be to have the same set of tests (current state) run against all the historic versions of tools we have collected (I suppose we can get them from conda ?).

While the single table that we have now is a good overview of a particular run, for the historic view, a couple of different views are also useful, such as a graph that shows tool-timestamp on the X-axis and passing tests on the y-axis. This would show how tools improve over time.

Immediate tasks would be to change the CSV format to have a one row per (tool, test) tuple instead of one row per test with columns representing the tools.

[wsnyder]

Seeing tool versions against head would be interesting.

Note my suggestion was to simply keep all old reports because they have been (ideally) analyzed by "both sides" - tools and tests. Running against "head tests" would not solve the original reason for the request which is problems int he other direction - there used to be passes against Verilator (#770) and Icarus (#732) that are broken due to test driver changes.

FWIW #446 is perhaps the best way (but more effort) to reduce test/driver problems. Though even if that is fixed I still think a simple history of reports has good value.

[hzeller]

So to see the reports we recorded, we could go through the history of the gh-pages:

git checkout gh-pages
git log --pretty=format:'<a type="text/html" href="https://raw.githubusercontent.com/SymbiFlow/sv-tests/%H/index.html">%cI (%cr)</a><br/>' > /tmp/list-historic-results.html

... then /tmp/list-historic-results.html will have a link to the old reports. HOWEVER, github does not want to encourage people serving HTML pages, so they don't set the mime-type to text/html :/ .. so clicking on any of the links show up as text in the browser.

So sending this to a local proxy that sets the mime-type correctly or maybe some javascript trickeries can help ?