Package CI candidates

[metagn]

This is a general issue for packages that should be in package CI until the situation has sufficiently improved, suggestions would be appreciated.

The release of 2.2 was delayed quite a bit by breaks in packages that were not tested in CI and had to be tested individually long after the breaks were introduced. The last time a package was added to CI was constantine in July 2024, which is somewhat recent, but the last ones before that were ssostrings and cowstrings in June 2023, before the release of 2.0. I can attest to constantine for example being the only package in CI which demonstrated some breaks when developing PRs like #24005. So it's long overdue for some updates.

Status

Individual libraries

Going off the list of packages which @narimiran made PRs to add 2.2 CI, not all are necessarily feasible for Nim CI:

• blscurve, currently in allowed failures due to missing setup for dependency blst
• confutils, had break Template param with default value nil and type untyped keeps the type untyped #24164
• kzg4844, C wrapper
• metrics
• nat-traversal
• secp256k1, C wrapper
• snappy, although supersnappy is tested
• ssz-serialization
• toml-serialization
• web3

Other status packages that may be worth adding:

• binary-serialization
• graphql
• minilru
• protobuf-serialization
• fork of lrucache

Except ssz-serialization, all of these have CI runs that last around 10 minutes at most (some don't have CI runs though). In the worst case, we don't have to run their entire test suites (as is done for stint).

Big packages

Most of the above libraries are dependencies of the bigger status toolchain repos, namely:

• nimbus-eth1
• nimbus-eth2
• eth, dependency of both
• libp2p, dependency of nimbus-eth2

These are the packages that catch the most regressions by far. It might not be necessary to test all of the ones above that are dependencies to these if we just test these.

These packages are quite a bit more difficult to test however, both due to the required setup and maintenance, as well as the potential increase to the CI time, since the full CI runs of these are 30 minutes or longer. Some options we have to deal with this are:

• Add a dedicated status batch similar to allowed_failures
• Don't run this batch for every single commit/PR: either allow manually triggering it for particularly impactful PRs, or run it on a schedule, like once a week. Or both.
• Do it in another repo similar to nightlies, ideally with support for CI'ing specific PRs to this repo.

This could be generalized to test any large toolchain using Nim.

Unfortunately I don't know much about the CI for these packages so I can't really go into specifics. Other opinions on this would be appreciated.

nim-lang packages

These have the advantage that we can manage & watch their CI, but it might still be useful to test them for PRs etc.

• nimble, compiler does not use HEAD version and so the HEAD version isn't tested, has own devel CI though
• nimlangserver, nimlsp is tested but langserver is on the nim-lang org and more actively developed, has own devel CI, caught Segfault in nimlangserver when -d:nimOptIter is defined #24094
  • see Package CI candidates #24266 (comment)
• threading, has own devel CI

Other community packages

Unfortunately again, as someone who mainly uses Nim to work on the Nim compiler, I'm not too aware of libraries that are important to people that don't interact with this repo. Off the top of my head, I can suggest:

• nimja or other template libraries, currently only templates is tested
• arturo which recently encountered Error: type mismatch: got <typedesc[A]>, but expected one of: A = proc (){.closure.} #23587, maybe min, spry if their code is not too old
• maybe chame, got nim check fails when checking chame #24288 although this wouldn't be caught by testing

Potentially harder to test:

• newer graphics libraries like owlkettle (GTK4 dependency, there is currently a problem linking to libraries in macos CI since update CI to macos 13 #24157), naylib
• newer server libraries in jester/httpbeast's abscence, like mummy; currently prologue is tested

Not all tested packages need to be popularly used, as long as they uniquely demonstrate a use of the language that is expected to keep working. Small, vetted macro libraries for example would be useful.

[jmgomez]

For nimlangserver we may want to pass --useSystemNim which recently got fixed in nimble v0.16.2 (will bump the koch version as soon as its released).
Speaking of nimble maybe its worth considering adding it to the list as well

[metagn]

I didn't even consider that nimble is not tested outside of the version the compiler installs. Added separate nim-lang packages section in case we are missing more of those.