[JOSS] Reproducibility - benchmarks do not run

[aholmes]

Regarding openjournals/joss-reviews#4991 - This repository is mentioned in the paper being reviewed under "Benchmarks." Upon cloning this repository and following the README.md to run the run_benchmark.py script, I receive this error:

$ python run_benchmark.py --verbose
===========================================
    Ackermann
===========================================
-------------------
    python
-------------------
['python3', '-m', 'timeit', '-s', 'from ackermann_mod import ackermann;import sys; sys.setrecursionlimit(3000);', 'ackermann(3,8)']
1 loop, best of 5: 326 msec per loop

-------------------
    pythran
-------------------
['time', 'pythran', '-O3', '-march=native', '-mtune=native', '-mavx', '-ffast-math', 'ackermann_mod.py']
Traceback (most recent call last):
  File "/home/aaron/pyccel-benchmarks/run_benchmark.py", line 238, in <module>
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'time'

Environment information:

$ python --version
Python 3.9.9

$ lsb_release -a
No LSB modules are available.
Distributor ID: Kali
Description:    Kali GNU/Linux Rolling
Release:        2020.4
Codename:       kali-rolling

$ uname -a
Linux AARON-XPS 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 GNU/Linux

[yguclu]

You do not have the time command on your Linux terminal?

[EmilyBourne]

I'm not sure what could be causing this, other than the time command not being available, but I thought that was available on most linux systems. Does PR #4 help with the issue by any chance ?

[aholmes]

time is only a shell built-in on my machines. From man bash:

   Pipelines
       A pipeline is a sequence of one or more commands separated by one of the control operators |  or  |&.   The
       format for a pipeline is:

              [time [-p]] [ ! ] command1 [ [|⎪|&] command2 ... ]

    ...

       If  the time reserved word precedes a pipeline, the elapsed as well as user and system time consumed by its
       execution are reported when the pipeline terminates.  The -p option changes the output format to that spec‐
       ified by POSIX.  When the shell is in posix mode, it does not recognize time as a reserved word if the next
       token begins with a `-'.  The TIMEFORMAT variable may be set to a format string that specifies how the tim‐
       ing information should be displayed; see the description of TIMEFORMAT under Shell Variables below.

Wikipedia has a brief explanation of this:

time(1) can exist as a standalone program (such as GNU time) or as a shell builtin in most case (e.g. in sh, bash, tcsh or in zsh).

Because subprocess attempts to execute time directly (that is, Python starts a new process that is time), and time is not a command on my machines (and likely most others), the process cannot start. subprocess does grant you the ability to start a shell instead - Python starts a process that is sh, and sh then starts the process desired (in this case, time), which will succeed in this case (I tested locally). However, shell=True has security implications that you may not want to expose your users to.

Looking around, I see a few alternatives that may work well here. This solution using the resource module looks promising. I will submit a PR shortly.