nanbield/x86_64: dynamic linker set to /lib64/ld-linux-x86-64.so.2

[rsflo]

Describe the bug
Executables that are cross-built using clang have the dynamic linker set to /lib64/ld-linux-x86-64.so.2 while it is actually found at /usr/lib/ld-linux-x86-64.so.2.
Build and and target are both x86_64. MACHINE is derived from poky's genericx86-64.
This is based on meta-clang commit 5170ec9 / branch nanbield (clang-17.0.4), poky tag yocto-4.3.1. meta-openembedded branch nanbield as of 19-DEC-2023.

To Reproduce
Steps to reproduce the behavior:

Build host:
$ bitbake meta-toolchain
$ "install SDK"

$ "source SDK environment file"
$ $CLANGCC -o ~/tmp/go ~/tmp/main.c
$ $READELF -p .interp ~/tmp/go
String dump of section '.interp':
[ 0] /lib64/ld-linux-x86-64.so.2

$ ls $OECORE_TARGET_SYSROOT
boot etc lib run usr var

$ ls $OECORE_TARGET_SYSROOT/lib
modules

ls $OECORE_TARGET_SYSROOT/usr/lib/ld*
[...]/usr/lib/ld-linux-x86-64.so.2

Target host:
$ /tmp/go
-bash: /tmp/go: cannot execute: required file not found

Expected behavior
Building an executable file using clang sets the dynamic linker to /usr/lib/ld-linux-x86-64.so.2

Additional context
Seems like the patches for Linux::getDynamicLinker() trip over the case the linker is neither found in /usr/lib64 nor in /lib and then defaults to /lib64/ld-linux-x86-64.so.2.
Affected patches:
0006-clang-driver-Check-sysroot-for-ldso-path.patch
0017-clang-driver-Add-dyld-prefix-when-checking-sysroot-f.patch

Previously that system was based on kirkstone and for the corresponding SDK target sysroot the linker (and a subset of shared libs) is still located in /usr and so it went fine. For nanbield all is put into /usr/lib.
(Building natively on target would probably work fine since a symlink from /usr to /usr/lib exists.)
While the latest state of master was not tried the set of relevant patches appear to be mostly identical.

There is a patch available for the mentioned ones, yet it covers x86_64 only. I'm willing to provide a complete one in case all of this makes sense :-)

0001-clang-set-x86_64-dyld-path-according-to-yocto-4.3.1.patch.txt

[rsflo]

Hi @kraj, I'd like to get your opinion on the way forward to this issue. Recently we've stumbled over a similar issue affecting gdb related to "usrmerge" (it cannot find shared libraries since it only searches in /lib of sysroot instead of /usr/lib).
It turned out that a Standard SDK (in Yocto parlance) provides the required symlinks as part of its sysroot, while a "reduced" one based on meta-toolchain does not -- which may explain why there are not too many complains in general :-)

A closer inspection shows that the functionality for creating the required symlinks is part of oe-core's image class and that does not apply when building the meta-toolchain recipe (which obviously is not an "image").
For now replicating that functionality in a local bbappend for meta-toolchain helps (having it commonly implemented in populate_sdk_base.bbclass would probably be a better fit) and so no longer the before mentioned patch to clang is required, while ideally I assume at least clang, gdb and likely gcc could get usrmerge-clean on its own.

Thanks

[kraj]

@rsflo I think using image based SDKs is recommended way for producing SDKs, for a bare mimimum SDK, using something like

bitbake -cpopulate_sdk core-image-minimal

should give you a small enough SDK comparable to what meta-toolchain produces.

[rsflo]

@kraj thanks for the suggestion. A new image recipe was created that is derived from core-image and this brings it down to almost the size of what meta-toolchain provides.

Other than that, is there general interest to get the aforementioned changes into meta-clang to not depend on the symlinks any longer?

[kraj]

@kraj thanks for the suggestion. A new image recipe was created that is derived from core-image and this brings it down to almost the size of what meta-toolchain provides.

Other than that, is there general interest to get the aforementioned changes into meta-clang to not depend on the symlinks any longer?

I am interested in solving this issue yes. So if you improve the proposed patch to include arm64/riscv64 into the consideration as well, secondly, building with multilib and without multilib cases, this would be more complete. Thirdly, we also need to ensure that we do not diverge from ABI for finding the ldso when usermerge is enabled.