Failures the first time a benchmark is run over a branch

[mmatera]

I am finding a systematic error when I run mathics-bench. After cleaning all, and running mathics-bench against the master,

cd mathics-core/ && make && make clean-cython && make clean && cd ..  >/dev/null 2>&1
mathics-bench -v -v -v <benchmark>

if I run

mathics-bench -v -v -v <benchmark>  <branch>

I get the following traceback:

...
    definition = Definition(
  File "/home/mauricio/Projects/mathics-benchmark/mathics-core/mathics/core/definitions.py", line 797, in __init__
    self.attributes = attributes
TypeError: 'int' object is not iterable

If now I run again

mathics-bench -v -v -v <benchmark>  <branch>

then the traceback disappears and the program runs normally. I get the same behavior if I run

cd mathics-core/ && make && make clean-cython && make clean && cd ..  >/dev/null 2>&1

before each call to mathics-bench. Since the last line self.attributes = attributes does not seem to raise the error TypeError: 'int' object is not iterable, I think that the problem is that we are not cleaning some python_cache files. @rocky @TiagoCavalcante Thoughts?

[TiagoCavalcante]

@mmatera seems it is not just the 1st time, but one time yes and one not. I guess mathics-core isn't being builded right. Checking the code.

[rocky]

I am finding a systematic error when I run mathics-bench. After cleaning all, and running mathics-bench against the master,

cd mathics-core/ && make && make clean-cython && make clean && cd ..  >/dev/null 2>&1
mathics-bench -v -v -v <benchmark>

Looks as though make clean in mathics-core isn't removing *pyc files and it should.

Someone want to add a PR for that? It is just a find like we do for cython but the pattern is pyc and pyo.

Pro-tip for debugging: use remake -x:

$ remake -x clean-cython && remake -x clean 
Reading makefiles...
Updating makefiles...
Updating goal targets...
 File 'clean-cython' does not exist.
Must remake target 'clean-cython'.
Makefile:62: target 'clean-cython' does not exist
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
find mathics -name "*.so" -type f -delete; \
find mathics -name "*.c" -type f -delete
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Successfully remade target file 'clean-cython'.
Reading makefiles...
Updating makefiles...
Updating goal targets...
 File 'clean' does not exist.
   File 'clean-cython' does not exist.
  Must remake target 'clean-cython'.
Makefile:62: target 'clean-cython' does not exist
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
find mathics -name "*.so" -type f -delete; \
find mathics -name "*.c" -type f -delete
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
  Successfully remade target file 'clean-cython'.
Must remake target 'clean'.
Makefile:67: update target 'clean' due to: clean-cython
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
for dir in mathics/doc ; do \
   (remake -C "$dir" clean); \
done; \
rm -f factorials || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Reading makefiles...
Updating makefiles...
Updating goal targets...
 File 'clean' does not exist.
Must remake target 'clean'.
remake[1]: Entering directory '/src/external-vcs/github/Mathics3/mathics-benchmark/mathics-core/mathics/doc'
Makefile:12: target 'clean' does not exist
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
remake -C tex clean
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Reading makefiles...
Updating makefiles...
Updating goal targets...
 File 'clean' does not exist.
Must remake target 'clean'.
remake[2]: Entering directory '/src/external-vcs/github/Mathics3/mathics-benchmark/mathics-core/mathics/doc/tex'
Makefile:44: target 'clean' does not exist
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
rm -f mathics.asy mathics.aux mathics.idx mathics.log mathics.mtc mathics.mtc* mathics.out mathics.toc || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
rm -f test-mathics.aux test-mathics.idx test-mathics.log test-mathics.mtc test-mathics.mtc* test-mathics.out test-mathics.toc || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
rm -f mathics.fdb_latexmk mathics.ilg mathics.ind mathics.maf mathics.pre || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
rm -f mathics_*.* || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
rm -f mathics-*.* documentation.tex /home/rocky/.local/var/mathics/doc_tex_data.pcl || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
rm -f mathics.pdf mathics.dvi test-mathics.pdf test-mathics.dvi || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
rm -f mathics-test.pdf mathics-test.dvi version-info.tex || true
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Successfully remade target file 'clean'.
remake[2]: Leaving directory '/src/external-vcs/github/Mathics3/mathics-benchmark/mathics-core/mathics/doc/tex'
Successfully remade target file 'clean'.
remake[1]: Leaving directory '/src/external-vcs/github/Mathics3/mathics-benchmark/mathics-core/mathics/doc'
...

[TiagoCavalcante]

@rocky thanks for finding it, tommorrow I'll create a PR.

[rocky]

One other thing we should be sensitive to is that the first time you run a Python program after cleaning, there is that step of saving the bytecode files. The might be an option in Python to indicate not to byte compile anything.

Easier though would be just to run the benchmark suite once setting the iterations to 1. That iteration would just ensure that anything that needs byte compiling is. And then run a second time to get the statistics.