Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Open
Pleune opened this issue Apr 16, 2024 · 20 comments
Open

Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Pleune opened this issue Apr 16, 2024 · 20 comments
Labels
bug Something isn't working

Comments

@Pleune
Copy link

Pleune commented Apr 16, 2024

Bug description

At this time, the WSL2/WSLg D3D12 hardware acceleration does not work out of the box due to library problems, even though the mesa.drivers package does have D3D12 support enabled by default.

I do not understand exactly what is going on. but there are a few ways to set LD_LIBRARY_PATH as a hack to fix it:

  • LD_LIBRARY_PATH=/run/opengl-driver/lib works with MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia but not Intel (My laptop has two GPUs)
  • LD_LIBRARY_PATH=ub:/run/opengl-driver/lib works with both Intel and Nvidia from above, where the ub folder has some basic library files copied from the WSL2 Ubuntu system - see below for contents

In both examples above, /run/opengl-driver/lib can be swapped with /usr/lib/wsl/lib. I'm not sure what the difference is here, other than that /usr/lib/wsl/lib has many more files.

I believe this could be two problems:

  1. the mesa.drivers files are not able to dlopen the WSLg files, since it has no way to know where they are.
  2. The Intel version of the D3D12 sub-driver-??? seems to need some additional libraries to link to that are not currently available.

To Reproduce

With the following config:


  programs.nix-ld = {
    enable = true;
    # I am using this nix-ld-rs too:
    #   package = nix-ld-rs.packages."${pkgs.system}".nix-ld-rs;
  };
  wsl = {
    enable = true;
    defaultUser = "${username}";
    nativeSystemd = true;
    useWindowsDriver = true;
    startMenuLaunchers = true;
  };

  hardware.opengl = {
    enable = true;
    driSupport = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      vaapiVdpau
    ];
  };

Then run any of the following commands:

glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=ub:/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=/usr/lib/wsl/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B

The ub folder I am using is: ub.tar.gz

Logs

Example of hardware acceleration working:

> LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B
name of display: :0
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: Some incorrect rendering might occur because the selected Vulkan device (Microsoft Direct3D12 (NVIDIA RTX A500 Laptop GPU)) doesn't support base Zink requirements: feats.features.logicOp have_EXT_custom_border_color have_EXT_line_rasterization
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Microsoft Corporation (0xffffffff)
    Device: D3D12 (Intel(R) Iris(R) Xe Graphics) (0xffffffff)
    Version: 24.0.3
    Accelerated: yes
    Video memory: 8146MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.1
    Max compat profile version: 4.1
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.0

< snip >

What happens without the LD_LIBRARY_PATH hack:

> glxinfo -B
name of display: :0
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0xffffffff)
    Device: llvmpipe (LLVM 17.0.6, 256 bits) (0xffffffff)
    Version: 24.0.3
    Accelerated: no
    Video memory: 7770MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 4.5
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2

< snip >

Where the ub folder has the following files:

> ls ub
libbsd.so.0
libedit.so.2
libmd.so.0
libstdc++.so.6
libtinfo.so.6
libz.so.1

WSL version

WSL version: 2.2.2.0
Kernel version: 5.15.150.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3296
@Pleune Pleune added the bug Something isn't working label Apr 16, 2024
@SuperSandro2000
Copy link
Member

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

Can you run the same commands with strace to see if it searches the right directories?

@Pleune
Copy link
Author

Pleune commented Apr 16, 2024

Here is the output of MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia strace -o strace.log glxinfo -B, where llvmpipe is used instead of the GPU

strace.log

I can see many references to /run/opengl-driver/lib here, so it must be searching that path. Maybe the initial d3d12 library is being loaded, but then when the code inside that runs, dlopen fails?

@SuperSandro2000
Copy link
Member

It is not searching for d3d12 in the right location.

openat(AT_FDCWD, "/etc/drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/mitch/.drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/vcxkk3l25hlix62aivl2prfhpczrvymp-expat-2.6.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zph9xw0drmq3rl2ik5slg0n2frw9lw5m-zlib-1.3.1/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/hqvkggqxqslgy2ahp1zswgh7im2kl5im-libxcb-1.16/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/qmy3imvwp6l129avxifnczw4p3fkdz0j-libX11-1.8.7/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zmpw448943piz8vdav28j8dqfy54xwjc-libxshmfence-1.3.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ss6gh67xv5jw6jh0l7dwmyx9823wvb60-zstd-1.5.5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/blw10rx1cayp2n2pkmyihpipifzgj2xq-wayland-1.22.0/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/1birlqjpkhsadz9zxa1in8z1apz7l1x4-systemd-minimal-libs-255.4/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ixi9ca45786dq510fk6h7crmi9259qcd-libdrm-2.4.120/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/c2yb135iv4maadia5f760b3xhbh6jh61-gcc-13.2.0-lib/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/5xylmnrx48vqlcfv1daj63kql91gflx3-mesa-24.0.3-drivers/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
futex(0x7fe8aa4b4e1c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
getdents64(4, 0x186db00 /* 4 entries */, 32768) = 144
getdents64(4, 0x186db00 /* 0 entries */, 32768) = 0
close(4)                                = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d/00-mesa-defaults.conf", O_RDONLY) = 4

@Pleune
Copy link
Author

Pleune commented Apr 16, 2024

hmm, libd3d12.so does exist in my nix store in from the wsl-lib and opengl-drivers packages:

> find /nix -name libd3d12.so
/nix/store/4mpn0189n0wrbi6xscricgsrdkx26bmx-wsl-lib/lib/libd3d12.so
/nix/store/gbfz16cazfd57f45654x916g06cwifxf-opengl-drivers/lib/libd3d12.so
/nix/store/1h8c6cx18mwn0i0kcz25mzbbf87mf8jp-opengl-drivers/lib/libd3d12.so
/nix/store/lk3pnpksmdw28m0nlj069vvg63l7ljjk-opengl-drivers/lib/libd3d12.so
/nix/store/0lxfb1j4yi3r0kaggq0hg0ad6q0i57ix-opengl-drivers/lib/libd3d12.so
/nix/store/57nb4gx6pmn46mvc8hlr26jmgm5j6i18-opengl-drivers/lib/libd3d12.so
/nix/store/6xq076h9n17fhb1w3f1pv1flspx1386m-opengl-drivers/lib/libd3d12.so
/nix/store/izbp5v8nvzs18gipcmd31iyywaqx10da-opengl-drivers/lib/libd3d12.so
/nix/store/v4wwzpgiazb98xjb0v4z0qpn4g2jwv84-opengl-drivers/lib/libd3d12.so
/nix/store/xzxjmcny8xdqv7b2sm0ws77zz65x9ylr-opengl-drivers/lib/libd3d12.so
/nix/store/10qydkiksz0zb9xn6fvlwr2s7swanzn1-opengl-drivers/lib/libd3d12.so
/nix/store/qq421zmhp9h5702mqlhirpxbf2jp5xb8-opengl-drivers/lib/libd3d12.so
/nix/store/hsd21kw4lpaxpivdyjrgz8c5vqm5iaq7-opengl-drivers/lib/libd3d12.so

I'm a good bit out of my depth here, but is there a way to add these packages as runtime dependencies for the mesa package?

Setting any of these directories as the LD_LIBRARY_PATH does allow at least the Nvidia d3d12 adapter to work

@Pleune
Copy link
Author

Pleune commented Apr 16, 2024

Thank you, you have given my enough information to discover the hardware.opengl.setLdLibraryPath = true; option, which makes the Nvidia driver work. Perhaps it would be best to set this as default.

For MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel, I need to also get libedit and libtinfo into /run/opengl-drivers somehow. Perhaps this is a NixOS bug and not a NixOS on WSL bug.

@Pleune
Copy link
Author

Pleune commented Apr 16, 2024

Alright, for a minimal configuration.nix only snippet, I have the following:

  hardware.opengl = {
    enable = true;
    driSupport = true;
    setLdLibraryPath = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      (libedit.overrideAttrs (attrs: {postInstall = (attrs.postInstall or "") + ''ln -s $out/lib/libedit.so $out/lib/libedit.so.2'';}))
    ];
  };

The libedit package only proides libedit.so.5 and libedit.so.0. d3d12 is looking for .2, so for the time being, this will work around that. It seems to pull in all of the other required library files as well.

@sjhaleprogrammer
Copy link

sjhaleprogrammer commented Jun 1, 2024

Traceback (most recent call last):
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 48, in <module>
    main()
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 39, in main
    prepare_environment()
  File "/home/samuel/git/stable-diffusion-webui/modules/launch_utils.py", line 386, in prepare_environment
    raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
(venv)

how do i get my nvidia gpu to work

@Parsifa1
Copy link

Parsifa1 commented Aug 2, 2024

Thank you, you have given my enough information to discover the hardware.opengl.setLdLibraryPath = true; option, which makes the Nvidia driver work. Perhaps it would be best to set this as default.

For MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel, I need to also get libedit and libtinfo into /run/opengl-drivers somehow. Perhaps this is a NixOS bug and not a NixOS on WSL bug.

It seems that the option hardware.opengl.setLdLibraryPath = true; no longer exists

@Pleune
Copy link
Author

Pleune commented Aug 12, 2024

@Parsifa1 Replace with

environment.sessionVariables.LD_LIBRARY_PATH = [ "/run/opengl-driver/lib" ];

@SuperSandro2000, what do you think about adding this to the repo by default, particularly since this easier option is removed now? It is still needed on my system for my GPU to function, and I also don't think it would cause anyone else any issues. I would be happy to submit a PR with more documentation and explanation if you think that would be proper.

Perhaps we could even add options to enable the other paths I have had to added for Cuda and whatnot to work. Below is the full environment section I have been using now:

environment.sessionVariables = {
  CUDA_PATH = "${pkgs.cudatoolkit}";
  EXTRA_LDFLAGS = "-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib";
  EXTRA_CCFLAGS = "-I/usr/include";
  LD_LIBRARY_PATH = [
    "/usr/lib/wsl/lib"
    "${pkgs.linuxPackages.nvidia_x11}/lib"
    "${pkgs.ncurses5}/lib"
    "/run/opengl-driver/lib"
  ];
  MESA_D3D12_DEFAULT_ADAPTER_NAME = "Nvidia";
};

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Aug 12, 2024

LD_LIBRARY_PATH is a big foot gun and can easily cause weird and hard to debug loading errors. I would love to really avoid it. If it is necessary we can add an option for it but I wouldn't turn it on by default.

The extra variables seem to be very user specific. I don't think they are a good fit.

@polybluez
Copy link

Maybe something like nixGL would be useful

@Pleune
Copy link
Author

Pleune commented Aug 15, 2024

Maybe something like nixGL would be useful

I have actually tried this and it does not really change anything.

As to Sandro's comments, Everything other than "/run/opengl-driver/lib" is for the windows/wsl nvidia drivers to work, and to support cuda properly. It would be better to include the two pkgs additions in another way, like symlinking them into the same opengl lib folder so they don't end up effecting other parts of the system, but I was not able to get that to work.

@polybluez
Copy link

polybluez commented Aug 15, 2024

Maybe something like nixGL would be useful

I have actually tried this and it does not really change anything.

No, I mean having an utility like nixGL for running an app with WSL DirectX drivers or even adding support for it in nixGL

@anticrisis
Copy link

anticrisis commented Aug 22, 2024

@SuperSandro2000 you said:

It is not searching for d3d12 in the right location.

I'm having the same ENOENT errors in strace when I launch emacs. I just installed NixOS on WSL in the past week or so, just trying to set everything up.

You also said:

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

but on my system, /usr/lib/wsl/lib is not symlinked. It contains all the cuda and d3d12 library files. And I apparently have no /run/opengl-driver/lib directory.

@Pleune
Copy link
Author

Pleune commented Aug 22, 2024

@SuperSandro2000 you said:

It is not searching for d3d12 in the right location.

I'm having the same ENOENT errors in strace when I launch emacs. I just installed NixOS on WSL in the past week or so, just trying to set everything up.

You also said:

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

but on my system, /usr/lib/wsl/lib is not symlinked. It contains all the cuda and d3d12 library files. And I apparently have no /run/opengl-driver/lib directory.

You may have to reboot your wsl system to get the run folder back. For some reason mine is still unreliable, particularly after a laptop sleep.

But I have been looking more into exactly what files are needed to be linked into to keep WSLg happy, without causing any ld hack issues. To settle onto a "perfect solution" I'm considering trying changing the symlink from wsl/lib into a copy and elfpatch instead. I've had some success doing this manually now, but have not had the time to properly add it to the repo config and try it out.

@anticrisis
Copy link

You may have to reboot your wsl system to get the run folder back. For some reason mine is still unreliable, particularly after a laptop sleep.

wsl --shutdown wasn't doing it, but a reboot of the machine, did bring back the /run/opengl-driver/lib/ directory.

@jacereda
Copy link

Are you setting this?

extraPackages = mkIf cfg.useWindowsDriver [

@anticrisis
Copy link

Are you setting this?

extraPackages = mkIf cfg.useWindowsDriver [

No! Seems quite important. I only read the documentation, I should have read the code. I will give that a go.

@cmspam
Copy link

cmspam commented Oct 22, 2024

Trying to get this working on AMD graphics. I have dual AMD/Nvidia. Nvidia is fine with the fixes here.
I've looked at strace but can't figure out what else to check to see what could be missing for AMD. Any advice to check?

@cmspam
Copy link

cmspam commented Oct 23, 2024

To fix it for radeon, I had to add the following to my configuration.nix

environment.sessionVariables = {
LD_LIBRARY_PATH = [
"${pkgs.openssl.out}/lib"
];
};

For some reason it requires libssl.so.3 to have radeon work. I wonder if this couldn't be done in a better way though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants