conbench tpch .... : 'failed': True, 'return_code': 0 (exit status should reflect error)

[jgehrcke]

Considering a command as 'failed' while its exit status is 0 -- that's a major maintenance burden.

'command': 'conbench tpch --iterations=3 --all=true --drop-caches=true --run-id=$RUN_ID --run-name="$RUN_NAME" --run-reason="$RUN_REASON"', 'started_at': '2023-06-11 17:24:35.418056', 'finished_at': '2023-06-11 19:50:58.723183', 'total_run_time': '2:26:23.305127', 'failed': True, 'return_code': 0

[alistaire47]

Do we know where failed and return_code are getting populated? I'm dealing with some related issues in arrowbench and labs/benchmarks where we need to catch errors, write/post results, and then reraise, but I'm not sure whether it will fix this or not

[austin3dickey]

I suspect here:

arrow-benchmarks-ci/buildkite/benchmark/run.py

Line 233 in 03717c4

return self.return_code != 0 or "ERROR" in self.stderr

[alistaire47]

Ok, so that's the labs/benchmarks exit status, I think. I'm messing with how it handles errors coming back from arrowbench, so I can take a look at how it exits; I know I saw at least one place where it sets a non-zero code, but apparently it's not everywhere it should

[jgehrcke]

Ok, so that's the labs/benchmarks exit status, I think.

Well, in the first msg here it shows that the command that exited with code zero was conbench tpch ...., so it's the legacy CLI. "The legacy CLI does not have a meaningful exit code" is maybe a better title for this issue :).

Exit code 0 should imply 100 % success, 0 errors. Sometimes the 'result complexity' cannot be reflected in a single number, but 0 should always mean 100 % success. Different errors can be reflected with different non-zero numbers. Looking for the word "error" in its stderr was probably a workaround around its non-authoritative (useless?) exit code.