Switch to rattler-build

[h-vetinari]

This PR is not immediately meant for merging, but for doing a logical separation of different changes on the way to exploratory emscripten support. Builds on #82

Still needs a bunch of fixes in our infra.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/recipe.yaml:

This is a rattler-build recipe and not yet lintable. We are working on it!

[h-vetinari]

There's a subtle behaviour change in how %LIBRARY_LIB% existed during the global build stage before, but doesn't anymore. No big deal, but triggered a failed copy here, c.f. 70e47af

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/recipe.yaml:

This is a v1 recipe and not yet lintable. We are working on it!

[wolfv]

I've re-triggered some builds to see if rattler-build 0.22.0 changes anything :)

[h-vetinari]

Interesting - on windows, cmake doesn't seem to be found; it's definitely there though, and also works before the translation to RB.

 │ │ Resolving build environment:
 │ │   Platform: win-64 [__win=0=0, __archspec=1=x86_64_v4]
 │ │   Channels: 
 │ │    - file:///D:/bld/
 │ │    - conda-forge
 │ │   Specs:
 │ │    - cmake
 │ │    - ninja
 │ │    - vs2019_win-64
 │ │    - vs_win-64
 │ │ [...]
 │ │ 'cmake' is not recognized as an internal or external command,
 │ │ operable program or batch file.

[wolfv]

We can see that the osx version is detected as 10.16 now, too:

 │ │ Resolving host environment:
 │ │   Platform: osx-64 [__unix=0=0, __osx=10.16=0, __archspec=1=skylake]
 │ │   Channels: 
 │ │    - file:///Users/runner/miniforge3/conda-bld/
 │ │    - conda-forge
 │ │   Specs:
 │ │    - __osx >=11.0

Which breaks the tests, since they require 11.0.

[wolfv]

Strange because jolt-physics seems fine (also CMake) conda-forge/jolt-physics-feedstock#3

[h-vetinari]

Strange because jolt-physics seems fine (also CMake) conda-forge/jolt-physics-feedstock#3

Perhaps another little 🐞 hiding in cache:? 🤔

[h-vetinari]

@conda-forge-admin, please rerender

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you but ran into some issues. Please check the output logs of the GitHub actions workflow below for more details. You can also ping conda-forge/core for further assistance or you can try [rerendering locally](https://conda-forge.org/docs/maintainer/updating_pkgs.html#rerendering-with-conda-smithy-locally.

The following suggestions might help debug any issues:

• Is the recipe/{meta.yaml,recipe.yaml} file valid?
• If there is a recipe/conda-build-config.yaml file in the feedstock make sure that it is compatible with the current global pinnnings.
• Is the fork used for this PR on an organization or user GitHub account? Automated rerendering via the webservices admin bot only works for user GitHub accounts.

_{This message was generated by GitHub actions workflow run https://github.com/conda-forge-webservices/actions/runs/11139428595.}

[h-vetinari]

FYI @wolfv @beckermr
The admin bot didn't manage to rerender here, but manually it worked (modulo changes necessary for conda-forge/conda-smithy#2057)

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found some lint.

Here's what I've got...

For recipe/recipe.yaml:

• The top level meta key cache is unexpected

[h-vetinari]

but manually it worked

Nevermind, that's almost certainly because I monkey-patched my local install (xref prefix-dev/rattler-build-conda-compat#56).

[h-vetinari]

@wolfv, cmake isn't found in any of the builds -- at least now the error is consistent across platforms I guess 😅

Looks like a cache: problem to me. Do you want me to open an issue in rattler-build?

[wolfv]

An issue with a small reproducer would be great :)

[h-vetinari]

Now it should be pretty minimal. ;-)

And now it's slightly less minimal, but actually reproduces the problem 😅

[h-vetinari]

@wolfv, thanks for the new rattler-build release! Looking better now (had to disable interactions with ci-setup, which cannot handle --experimental yet either). There was a segfault on linux-aarch64 that went away on restart, but otherwise unix looks fine.

On win-64 the tests fail with some unclear cause

 │ │ │ Installing test environment
 │ │ │  Successfully updated the test environment
 │ │ │ Testing commands:
 │ │ │ (base) C:\Users\VSSADM~1\AppData\Local\Temp\.tmpdO7zDL>IF "" == "" (call C:\Users\VSSADM~1\AppData\Local\Temp\.tmpdO7zDL\build_env.bat ) 
 │ │ │
 │ │ ╰─────────────────── (took 0 seconds)
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 71 seconds)
 × error Error building package: failed to run test: Script failed with status Some(1).
 × error Work directory: "C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\.tmpdO7zDL"
 × error To debug the build, run it manually in the work directory (execute the `./conda_build.sh` or `conda_build.bat` script)
Error:   × failed to run test: Script failed with status Some(1).

and on win-arm64 it crashes as before, but now we see why

thread 'main' panicked at src\cache.rs:281:9:
assertion `left == right` failed: Prefixes must have the same length: "D:\\bld\\bld\\rattler-build_libzlib_1727940322\\h_env" != "D:\\bld\\bld\\rattler-build_libzlib-wapi_1727940322\\h_env"
  left: 49
 right: 54

Difference is related to the outputs somehow, i.e. rattler-build_libzlib{,-wapi}_1727940322 in the environment path.

[wolfv]

Yes, awesome. Thanks for retrying! On win-64 it fails during test-execution if I see correctly. The win-arm64 issue is curious. Will have to investigate (but looks fixable!).

[h-vetinari]

On win-64 it fails during test-execution if I see correctly.

Is this how test failures generally look, or are we running into specific case here? If it's due to a failing test (and not some problem in the supporting infra), I'd want to have much more information about which line or assertion failed, rather than Some(1).

The win-arm64 issue is curious. Will have to investigate (but looks fixable!).

Cool, thanks!

[wolfv]

@h-vetinari if you have an idea about a different way of invoking cmd.exe that would be welcome :) Maybe there are some parameters that will print the line that is executed etc. (like bash).

[h-vetinari]

@h-vetinari if you have an idea about a different way of invoking cmd.exe that would be welcome :) Maybe there are some parameters that will print the line that is executed etc. (like bash).

Just execute the individual instructions from the test section line-by-line?

[wolfv]

I don't think that's the best solution as that's pretty different to other script executions. If it's in a list then it would be possible (otherwise we'd have to parse the script and split it on logical boundaries). I'll try to see if there is a way to enable a better way to trace script execution.

Btw. for package content tests, you can use the new package_content test that produce much better output and run insanely fast (because they only check the paths.json file) :)

Example: https://github.com/conda-forge/jolt-physics-feedstock/blob/be2718fdc372f0bd5cb33950751eb7eaf092a591/recipe/recipe.yaml#L54-L59

[h-vetinari]

I don't think that's the best solution as that's pretty different to other script executions.

I think this is what conda-build does (but don't know); I'm not talking about the content of some .cmd or .bat script by the way, just the stuff that's under:

test:
  commands:
    - ...

should be executed one-by-one (and that principle can even apply to all platforms), because you really want to know at which line things failed, and not just "in the test section".

Btw. for package content tests, you can use the new package_content test that produce much better output

Sure, that'll cover most cases I hope. But wherever we run things that cannot be translated to other tests under commands:, those lines should be executed separately. They're already a list when they come out of the yaml, so you'd just have to iterate through.

[h-vetinari]

you can use the new package_content test

It also doesn't work here because we have a lib that doesn't follow the standard naming pattern (z.lib, zlib.lib, zlib.dll, zlibstatic.lib), and more importantly, the DLL and the import library are in separate outputs, and also there's a DLL in a non-standard location (not %LIBRARY_BIN%).

[wolfv]

You can still use globs to check etc. make them platform independent and so on. All these cases are easily covered.

[wolfv]

https://prefix-dev.github.io/rattler-build/latest/reference/recipe_file/#check-for-package-contents

[h-vetinari]

Thanks, I guess using files: directly will work for presence tests.

You can still use globs to check etc. make them platform independent and so on. All these cases are easily covered.

However, we often want to specifically check the absence of something, mostly that there's no static library. I don't see how that's covered yet.

[wolfv]

Yes, we already have the design for negative globs as well :) prefix-dev/rattler-build#915

[h-vetinari]

I'll try to switch to the package_content tests, but regardless of how much can be converted, having no output for

test:
  commands:
    - ...

is a complete dealbreaker from my POV. I don't even see the inconsistency you mentioned with other script executions (well, those that are defined in a yaml list at least) - in those cases it's equally important to know which line failed.

[wolfv]

Definitely! If you have a simple repro I can debug or can come up with something myself