kraft fails to build arm64 images in container environment

[nderjung]

The kraft container environment is the environment where all the tools are provided for building unikraft unikernels, including the kraft program itself. W.r.t. this issue, it also includes baked compiled version of gcc for Unikraft's supported architectures. However, when targetting the arm64 architecture, the as program fails to compile, yielding the following error:

make: Entering directory '/tmp/build/80754af9/built/unikraft'
CP      config
GEN     libsyscall_shim: syscall_map.h.new
GEN     libsyscall_shim: syscall_stubs.h.new
GEN     libsyscall_shim: syscall_nrs.h.new
GEN     libsyscall_shim: syscall_nrs2.h.new
GEN     libsyscall_shim: provided_syscalls.h.in.new
GEN     libsyscall_shim: provided_syscalls.h
GEN     libsyscall_shim: uk_syscall.c
GEN     libsyscall_shim: uk_syscall6.c
GEN     libsyscall_shim: uk_syscall_r.c
GEN     libsyscall_shim: uk_syscall6_r.c
GEN     libsyscall_shim: uk_syscall_r_fn.c
GEN     libsyscall_shim: uk_syscall_name.c
GEN     libsyscall_shim: uk_syscall_name_p.c
GEN     libsyscall_shim: libc_stubs.c
make: Leaving directory '/tmp/build/80754af9/built/unikraft'
GEN     libsyscall_shim: syscall_map.h.new
GEN     libsyscall_shim: syscall_stubs.h.new
GEN     libsyscall_shim: syscall_nrs.h.new
GEN     libsyscall_shim: syscall_nrs2.h.new
GEN     libsyscall_shim: provided_syscalls.h.in.new
CC      libkvmplat: pl011.common.o
as: unrecognized option '-EL'
make[1]: *** [/tmp/build/80754af9/built/unikraft/support/build/Makefile.build:50: /tmp/build/80754af9/built/apps/helloworld/build/libkvmplat/pl011.common.o] Error 1
make: *** [Makefile:990: sub-make] Error 2

To reproduce locally, build the container environment:

make docker-qemu
make UK_ARCH=x86_64 TARGET=gcc-build docker-gcc
make UK_ARCH=arm64 TARGET=gcc-build docker-gcc
make UK_ARCH=arm TARGET=gcc-build docker-gcc
make docker-kraft

Then use the container environment to build an arm64 image:

docker run -it --rm -v /tmp/test-unikernel:/usr/src/unikraft --entrypoint bash --device /dev/kvm unikraft/kraft:staging
# you are now inside a container
kraft list update
kraft list pull helloworld@staging
cd /usr/src/unikraft/apps/helloworld
kraft configure
# select arm64
kraft build

My initial thoughts, without having dived deeply into this problem, that it is to do with the arm64 libraries in the container, whether they are symbollically linked correctly or their paths are not correct.

[hisharanwithu]

The error is related to the way aarch64 gcc toolchain is built and packaged into the container. We miss taking in some of the necessary file from directories like aarch64-linux-gnu-*/. One way to fix this would be while performing configure set the correct install path and let the gcc build system to copy the necessary files to the expected thereby we don't miss any files.

[razvanvirtan]

Apparently the toolchain built in the container is always trying to use /bin/as as the assembler, even if we try to compile an ARM64 image. This causes that -EL error.
I've tried to link /bin/as to aarch64-linux-gnu-as (just for testing) and the -EL error disappeared. However, the following error was generated at linking: ld: cannot open linker script file ldscripts/aarch64elf.xr: No such file or directory. A detailed description of this issue can be found here: https://www.labbott.name/blog/2018/07/10/the-cabbage-patch-for-linker-scripts/

My proposal would be to replace the current ARM64 toolchain with the linaro one. I've tried that in the container and kraft build is now running without any problems.We are also using linaro for cross-compiling at one of our classes in the university and it is the toolchain installed on the physical host provided for us by ARM.

[nderjung]

Ok cool, thanks for checking this out!

Could you put together a PR together with fixes and also the linaro toolchain which integrates into the build process of the kraft container?

Probably the linaro toolchain needs its own Dockerfile, an appropriate target which catches the target arch so builds the right image

[razvanvirtan]

Sure, I'll find a way to integrate it.
Also, we will need to fix the condition here: https://github.com/unikraft/kraft/blob/29c6fba81b885d5fb3b5d2dc76b976644828df30/kraft/plat/runner/kvm.py#L70
because trying to enable hardware acceleration for ARM64 on a x86 host will cause kraft run to fail.

[nderjung]

So this comment needs to be uncommented?

[razvanvirtan]

Yes, and we should add import platform at the beginning of kvm.py. I'll come with a PR after I integrate the linaro toolchain in the container configuration.