Dates and Deadlines

[vtjeng]

@ttj, is there a centralized resource for all dates and deadlines (e.g. NeurIPS) related to vnn-comp? It would be helpful to have one including all important milestones (even if the dates are not yet set in stone) for planning purposes.

[ttj]

The website has dates:

https://sites.google.com/view/vnn20/vnncomp

The hard deadline is for the workshop (July 19) where we'll discuss the results of this, so it would be ideal to have everything finalized at least a week before that so presentations, etc. can get finalized.

We can breakdown further into what's expected (mentioned in the various discussion threads, basically a Dockerfile and reproducible script to run the benchmarks) and will add some further detail soon, but please let us know of any specific questions.

[stanleybak]

Should we be running things locally to get timing results, or will the results be what you run from the docker files? what are the specs of the machine running the docker measurements? Can we test on it?

[ttj]

Should we be running things locally to get timing results, or will the results be what you run from the docker files? what are the specs of the machine running the docker measurements? Can we test on it?

Yes, just run locally and report the machine characteristics. For repeatability, we'll just re-run as a basic check. In the future, we'll try to have common infrastructure, etc., but that won't realistically happen this iteration.

[ttj]

Note that the date of the workshop shifted slightly to July 21, the program is available here:

https://sites.google.com/view/vnn20/program

If you plan to attend the live/synchronous aspects, please register through the main CAV registration (free) and indicate VNN as one of the interested workshops (you may also be interested in FOMLAS):

http://i-cav.org/2020/attending/

[FlorianJaeckle]

Hi @ttj,
Are there any deadlines for submitting our results apart from the final July 21 one?
We were also wondering how we should present our results? Should we submit a written report along with the result tables and plots?

[ttj]

Are there any deadlines for submitting our results apart from the final July 21 one?
We were also wondering how we should present our results? Should we submit a written report along with the result tables and plots?

Please try to have things done by this Friday, July 17, but very latest Monday, July 20 (at a reasonable time, let's say noon US central time) so we'll have some time to make the presentation for the July 21 workshop.

For the results report, I will share the overleaf link soon, we can finish the report after the workshop, but the workshop presentation will only include results submitted by July 20 as otherwise we won't have time to integrate them. Tables/plots are great for now, the easier to integrate into a presentation, the better. :-)

[vtjeng]

@ttj is there a standard format that we should submit the results in? for now, I've been doing CSV files; a sample for the ACASXU dataset from #2 can be found below:

(The relevant values that would be shared across different verification tools are PropertyID, NetworkID, VerificationResult, TotalTime).

PropertyID,NetworkID,VerificationResult,InternalSolveStatus,ObjectiveValue,ObjectiveBound,InputValue,OutputValue,MainSolveTime,TotalTime
1,1_1,Timeout,UserLimit,-4.0112476887892345,68.57404491846057,NaN,NaN,300.01608085632324,308.500427861
2,1_1,Timeout,UserLimit,-0.0030905613

[ttj]

@ttj is there a standard format that we should submit the results in? for now, I've been doing CSV files; a sample for the ACASXU dataset from #2 can be found below:

Thanks! This looks great in general, as long as it's easy to generate e.g. figures/tables from to put in the report/presentation, however, let's move this discussion to #8 where I'll respond more