Attempt to kill bg process in tests

[grarco]

Describe your changes

Some e2e tests rely on background processes, especially the IBC ones with hermes and gaiad. There are cases in which the test fails and some of these tasks (especially gaiad) are not affected by the crash of the Namada node. In this case the runtime drops the NamadaBgCmd but unfortunately dropping a JoinHandle only detaches the target task. This is undesired cause if we run the test again the conflicting instances of the background process will cause another test failure.

This PR attempts to run some cleanup code when dropping NamadaBgCmd to kill the task.

Checklist before merging

• If this PR has some consensus breaking changes, I added the corresponding breaking:: labels
  • This will require 2 reviewers to approve the changes
• If this PR requires changes to the docs or specs, a corresponding PR is opened in the namada-docs repo
  • Relevant PR if applies:
• If this PR affects services such as namada-indexer or namada-masp-indexer, a corresponding PR is opened in that repo
  • Relevant PR if applies:

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.24%. Comparing base (45e7e90) to head (1773a17).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3894   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files         341      341           
  Lines      105240   105240           
=======================================
+ Hits        77085    77087    +2     
+ Misses      28155    28153    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sug0]

unrelated to this PR, but this try_recv might be contributing substantially to the resource usage of e2e tests :P

[grarco]

I guess the quick fix would be to call recv_timeout instead with a sane timeout. Probably a better solution would be to switch to async channels?

[sug0]

I think async channels wouldn't help. there is no event here triggering a new channel read, so the only reasonable thing would be what you suggested first: a poll with a timer

[grarco]

Makes sense, I've opened #3897 to track that, I'll merge this one, thank you!