Android API 23, Runtime Crash in .NET 9 #106025

jonathanpeppers · 2024-08-06T15:57:06Z

Description

We have a nightly test pipeline that runs our test suite on older Android emulators.

The API 23 emulator crashes at runtime with:

08-06 03:33:12.480  2745  2745 F mono-rt : [ERROR] FATAL UNHANDLED EXCEPTION: Nested exception detected.
08-06 03:33:12.480  2745  2745 F mono-rt : Original Exception: at System.Threading.Thread.get_CurrentThread () [0x00009] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Threading.Monitor.Exit (object) [0x0000e] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.AppContext.GetData (string) [0x00037] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.AppContext.TryGetSwitch (string,bool&) [0x00040] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.AppContextConfigHelper.GetBooleanConfig (string,bool) [0x00000] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.LocalAppContextSwitches.GetDefaultShowILOffsetSetting () [0x00014] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Diagnostics.StackTrace.ToString (System.Diagnostics.StackTrace/TraceFormat,System.Text.StringBuilder) [0x002ee] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Diagnostics.StackTrace.ToString (System.Diagnostics.StackTrace/TraceFormat) [0x0000b] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.GetStackTrace () [0x00007] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.get_StackTrace () [0x00023] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.ToString () [0x0001f] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.ToString () [0x0000c] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.ToString () [0x0000c] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : 
08-06 03:33:12.480  2745  2745 F mono-rt : Nested exception:at System.Threading.Thread.get_CurrentThread () [0x00009] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Threading.Monitor.Exit (object) [0x0000e] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.AppContext.GetData (string) [0x00037] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.AppContext.TryGetSwitch (string,bool&) [0x00040] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.AppContextConfigHelper.GetBooleanConfig (string,bool) [0x00000] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.LocalAppContextSwitches.GetDefaultShowILOffsetSetting () [0x00014] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Diagnostics.StackTrace.ToString (System.Diagnostics.StackTrace/TraceFormat,System.Text.StringBuilder) [0x002ee] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Diagnostics.StackTrace.ToString (System.Diagnostics.StackTrace/TraceFormat) [0x0000b] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.GetStackTrace () [0x00007] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.get_StackTrace () [0x00023] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.ToString () [0x0001f] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.ToString () [0x0000c] in <d7679677bb2b4cc291070c60be6bf246>:0
08-06 03:33:12.480  2745  2745 F mono-rt : at System.Exception.ToString () [0x0000c] in <d7679677bb2b4cc291070c60be6bf246>:0

This appears to be working fine on newer API levels.

Reproduction Steps

Run this test project on an API 23 emulator:

https://github.com/dotnet/android/blob/main/tests/Mono.Android-Tests/Runtime-Microsoft.Android.Sdk/Mono.Android.NET-Tests.csproj

Expected behavior

Our test suite passes on API 23 emulators.

Actual behavior

Our test suite crashes on API 23 emulators.

Regression?

Yes, this is working in .NET 8.

Known Workarounds

n/a

Configuration

Noticed with:

.NET SDK 9.0.100-rc.1.24381.10
Runtime 9.0.0-rc.1.24380.7

I think this was also happening in .NET 9 Preview 7 builds.

Other information

logcat-Release23-Mono.Android.NET_Tests.txt

The text was updated successfully, but these errors were encountered:

dotnet-policy-service · 2024-08-06T19:57:28Z

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

akoeplinger · 2024-08-06T19:58:22Z

This looks like two issues in one: it's trying to throw an exception but then crashes because of the nested exception.

dotnet-policy-service · 2024-08-09T16:29:21Z

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

dotnet-policy-service · 2024-08-09T16:29:21Z

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

steveisok · 2024-08-09T16:56:30Z

@jonathanpeppers from the logcat, it appears libSystem.Native could not be loaded. Is there something wrong with your setup?

jonathanpeppers · 2024-08-09T17:11:36Z

@grendello do you know what might cause:

08-06 03:33:11.720  2745  2745 D monodroid-assembly: monodroid_dlopen: hash for name 'libSystem.Native' is 0x5b9ade60
08-06 03:33:11.721  2745  2745 D monodroid-assembly: monodroid_dlopen: hash match found, DSO name is 'libSystem.Native.so'
08-06 03:33:11.723  2745  2745 I monodroid-assembly: Trying to load shared library '/data/app/Mono.Android.NET_Tests-1/base.apk!/lib/x86/libSystem.Native.so'
08-06 03:33:11.732  2745  2745 I monodroid-assembly: Failed to load shared library '/data/app/Mono.Android.NET_Tests-1/base.apk!/lib/x86/libSystem.Native.so'. dlopen failed: library "/data/app/Mono.Android.NET_Tests-1/base.apk!/lib/x86/libSystem.Native.so" not found
08-06 03:33:11.732  2745  2745 I monodroid-assembly: Trying to load shared library '/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk!/lib/x86/libSystem.Native.so'
08-06 03:33:11.735  2745  2745 I monodroid-assembly: Failed to load shared library '/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk!/lib/x86/libSystem.Native.so'. dlopen failed: cannot find "libc.so" from verneed[1] in DT_NEEDED list for "/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk"
08-06 03:33:11.736  2745  2745 I monodroid-assembly: Trying to load shared library '/data/app/Mono.Android.NET_Tests-1/base.apk!/lib/x86/libSystem.Native'
08-06 03:33:11.736  2745  2745 I monodroid-assembly: Failed to load shared library '/data/app/Mono.Android.NET_Tests-1/base.apk!/lib/x86/libSystem.Native'. dlopen failed: library "/data/app/Mono.Android.NET_Tests-1/base.apk!/lib/x86/libSystem.Native" not found
08-06 03:33:11.737  2745  2745 I monodroid-assembly: Trying to load shared library '/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk!/lib/x86/libSystem.Native'
08-06 03:33:11.737  2745  2745 I monodroid-assembly: Failed to load shared library '/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk!/lib/x86/libSystem.Native'. dlopen failed: library "/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk!/lib/x86/libSystem.Native" not found
08-06 03:33:11.738  2745  2745 W monodroid-assembly: Shared library 'libSystem.Native' not loaded, p/invoke 'SystemNative_SchedGetCpu' may fail
08-06 03:33:11.738  2745  2745 F monodroid-assembly: Failed to load symbol 'SystemNative_SchedGetCpu' from shared library 'libSystem.Native'

But earlier in the log:

08-06 03:33:11.008  2745  2745 D monodroid-assembly: /data/app/Mono.Android.NET_Tests-1/split_config.x86.apk entry: lib/x86/libSystem.Native.so
08-06 03:33:11.008  2745  2745 D monodroid-assembly:     ZIP: local header offset: 921492; data offset: 933888; file size: 96760
08-06 03:33:11.008  2745  2745 D monodroid-assembly: Found a shared library entry lib/x86/libSystem.Native.so (index: 2; name: libSystem.Native.so; hash: 0x79d6a0ba; apk offset: 0)

This is a 32-bit emulator, could this be fallout of the 16k alignment changes?

grendello · 2024-08-19T13:30:40Z

@jonathanpeppers 32-bit shared libraries are aligned to 4k, 16k applies only to 64-bit targets. That's what NDK r27 does, so I followed suit, but it might be still a system error. We store the dlerror message when loading the shared library and pass it back to Mono via the dlopen hook here.

@steveisok @akoeplinger does Mono log the err when it sees one?

akoeplinger · 2024-08-19T15:13:25Z

No, looks like we don't do anything with the error:

runtime/src/mono/mono/utils/mono-dl.c

Lines 269 to 271 in dec4568

    
           char *error_msg = NULL; 
        
           lib = handler->load_func (name, lflags, &error_msg, handler->user_data); 
        
           g_free (error_msg);

akoeplinger · 2024-08-19T15:18:41Z

@grendello isn't that the real error? sounds weird though

dlopen failed: cannot find "libc.so" from verneed[1] in DT_NEEDED list for "/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk"

grendello · 2024-08-19T15:33:42Z

@akoeplinger on one hand yes, it is, on the other... how do we even run the app if libc.so cannot be found? :) Just doesn't make sense

BrzVlad · 2024-09-17T10:11:48Z

Can't reproduce this. Tried running on my apple chip with arm64 AVD, failed to reproduce, then noticed that in the report the device was android-x86. I then tried the following:

on my intel macbook, I created a Pixel 5 AVD with the following system image Android 6.0, API Level 23, x86
cloned dotnet/android with recent main checked out
make prepare; make jenkins
./dotnet-local.sh build -t:RunTestApp tests/Mono.Android-Tests/Runtime-Microsoft.Android.Sdk/Mono.Android.NET-Tests.csproj /p:RuntimeIdentifier=android-x86
logcat output: 09-17 12:59:39.991 4617 4632 I NUnit : Passed: 859, Failed: 0, Skipped: 7, Inconclusive: 0, Total: 0, Filtered: 0

jonathanpeppers · 2024-09-17T13:35:53Z

Our nightly CI is failing, because we dropped 32-bit by default:

[BT The app doesn't support ABI architectures of the device. Device ABIs: [x86], app ABIs: [arm64-v8a, x86_64].

Let me fix these tests and find out if latest is working.

jonathanpeppers · 2024-09-17T14:08:49Z

Fixing our tests here:

[tests] fix nightly APK tests for 32-bit emulators android#9308

This should find out if this is still an issue with recent-ish RC 2.

jonathanpeppers · 2024-09-17T17:09:07Z

@BrzVlad I think I'm still seeing that libSystem.Native.so not loading might be the underlying issue:

09-17 14:41:32.839  2758  2758 I monodroid-assembly: Failed to load shared library '/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk!/lib/x86/libSystem.Native.so'. dlopen failed: cannot find "libc.so" from verneed[1] in DT_NEEDED list for "/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk"

It looks like this works on API 21 and fails on API 23? And then x86 is the only emulator option for those old Android versions.

jeffhandley · 2024-10-08T04:38:00Z

@jonathanpeppers / @steveisok -- Checking in on this. Does it still need resolution before .NET 9 GA? If so, who should be assigned to bring it to closure?

jonathanpeppers · 2024-10-14T13:50:15Z

It looks like this combination isn't working:

libSystem.Native.so cannot be loaded on...
x86 (this is the only emulator option, for older Android versions they don't have x86_64)
API 23 or older

So, it's unknown how impacting this will be for customers. It's hard to know if you would see the same thing on arm64 devices.

jeffhandley · 2024-10-16T22:24:50Z

@jonathanpeppers -- @tannergooding pointed out that API 23 and older appear to be completely out of support by Google now.

See https://developer.android.com/google/play/requirements/target-sdk:

New apps and app updates must target Android 14 (API level 34) or higher to be submitted to Google Play; except for Wear OS and Android TV apps, which must target Android 13 (API level 33) or higher.

Existing apps must target Android 13 (API level 33) or higher to remain available to new users on devices running Android OS higher than your app's target API level. Apps that target Android 12 (API level 31) or lower (Android 10 (API level 29) or lower for Wear OS and Android 11 (API level 30) or lower for Android TV), will only be available on devices running Android OS that are the same or lower than your app's target API level.

I'm inclined to close this out based on that info. What do you think?

tannergooding · 2024-10-16T22:32:46Z

Notably I'm not sure its "completely out of support" but rather that it isn't supported if you're wanting to target the "latest devices".

There's still reasons to target downlevel API levels to support some older devices, but there's an overall balance required and I don't think an official page for what versions are supported across devices still exist.

jonathanpeppers · 2024-10-16T23:22:53Z

The “target” is what API you compiled against, Google Play requires very recent for this. “Minimum” API level is what you can run on. .NET has a minimum of API 21.

steveisok · 2024-10-17T00:54:38Z

It looks like this combination isn't working:

libSystem.Native.so cannot be loaded on...

x86 (this is the only emulator option, for older Android versions they don't have x86_64)

API 23 or older

So, it's unknown how impacting this will be for customers. It's hard to know if you would see the same thing on arm64 devices.

Based on @BrzVlad's previous analysis, are we sure this is a runtime issue?

jonathanpeppers · 2024-10-17T15:29:21Z

We are checking if a newer version of the API 23 emulator solves it:

[tests] use x86_64 API 23 image on nightly pipeline android#9414

Does something test "hello world" on Android on an API 23 (or 21) emulator? Is there a Helix lane for this?

If you have that running somewhere, and it would load libSystem.Native.so we can close this.

matouskozak · 2024-10-17T16:56:17Z

We are checking if a newer version of the API 23 emulator solves it:

[tests] run sdkmanager update before installing emulators android#9414

Does something test "hello world" on Android on an API 23 (or 21) emulator? Is there a Helix lane for this?

If you have that running somewhere, and it would load libSystem.Native.so we can close this.

There are Helix queues with 21/23 emulators, ubuntu.2204.amd64.android.[21/23] that you can use. By default, we only run tests on API 29 though.

jonathanpeppers · 2024-10-17T17:44:40Z

@matouskozak if one of those is succeeding with latest .NET 9 builds we can close this.

If you can find a build, would there be a log message saying libSystem.Native.so is successfully loaded?

matouskozak · 2024-10-18T06:50:58Z

@matouskozak if one of those is succeeding with latest .NET 9 builds we can close this.

If you can find a build, would there be a log message saying libSystem.Native.so is successfully loaded?

Does this issue need Android SDK to reproduce? If so, I think we don't run any tests with the dotnet/android code inside runtime.

We did run a .NET 9 runtime validation on Android API 21 - 32 (#106951) and I don't recall any issues with loading libSystem.Native.so. However, it is possible we could had missed this as there was a lot of flakiness on the older Android APIs (21 and 22 especially).

@jonathanpeppers if there is a repro only using dotnet/runtime I can setup a PR with older Android emulators to test it. Or if dotnet/android can use Helix we could open the PR there.

jonathanpeppers · 2024-10-18T13:54:37Z

After more investigation, we think this is some kind of bug in the x86, API 23 emulator image. API 21 is also working on our same pipeline, so this covers our minimum supported devices. API 23 was when Android first introduced some changes to loading native libraries, and its likely buggy.

We can close this, as we don't think it's a runtime issue.

There is some conversation on discord here for details:

https://discord.com/channels/732297728826277939/732297837953679412/1296764055654240317

Context: dotnet/runtime#106025 Context: https://discord.com/channels/732297728826277939/732297837953679412/1296764055654240317 `Mono.Android-Tests` on API 23 crash on launch with: 09-17 14:41:32.839 2758 2758 I monodroid-assembly: Failed to load shared library '/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk!/lib/x86/libSystem.Native.so'. dlopen failed: cannot find "libc.so" from verneed[1] in DT_NEEDED list for "/data/app/Mono.Android.NET_Tests-1/split_config.x86.apk" But this same library appears to be correct, and loads on all of: * x86 API 21 * x86_64 API 23 * x86 API 24 This leads us to believe there is some kind of bug with loading native libraries on the API 23 x86 emulator image. For now, let's just target x86_64 on API 23 for the time being.

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Aug 6, 2024

dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Aug 6, 2024

akoeplinger added area-System.Runtime os-android runtime-mono specific to the Mono runtime and removed untriaged New issue has not been triaged by the area owner needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Aug 6, 2024

akoeplinger added this to the 9.0.0 milestone Aug 6, 2024

akoeplinger added os-android and removed os-android labels Aug 9, 2024

jonathanpeppers mentioned this issue Sep 17, 2024

[tests] fix nightly APK tests for 32-bit emulators dotnet/android#9308

Merged

jonathanpeppers closed this as completed Oct 18, 2024

jonathanpeppers mentioned this issue Oct 18, 2024

[tests] use x86_64 API 23 image on nightly pipeline dotnet/android#9414

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Android API 23, Runtime Crash in .NET 9 #106025

Android API 23, Runtime Crash in .NET 9 #106025

jonathanpeppers commented Aug 6, 2024

dotnet-policy-service bot commented Aug 6, 2024

akoeplinger commented Aug 6, 2024

dotnet-policy-service bot commented Aug 9, 2024

dotnet-policy-service bot commented Aug 9, 2024

steveisok commented Aug 9, 2024

jonathanpeppers commented Aug 9, 2024

grendello commented Aug 19, 2024

akoeplinger commented Aug 19, 2024

akoeplinger commented Aug 19, 2024 •

edited

Loading

grendello commented Aug 19, 2024 •

edited

Loading

BrzVlad commented Sep 17, 2024 •

edited

Loading

jonathanpeppers commented Sep 17, 2024

jonathanpeppers commented Sep 17, 2024

jonathanpeppers commented Sep 17, 2024

jeffhandley commented Oct 8, 2024

jonathanpeppers commented Oct 14, 2024

jeffhandley commented Oct 16, 2024

tannergooding commented Oct 16, 2024

jonathanpeppers commented Oct 16, 2024

steveisok commented Oct 17, 2024

jonathanpeppers commented Oct 17, 2024 •

edited

Loading

matouskozak commented Oct 17, 2024

jonathanpeppers commented Oct 17, 2024

matouskozak commented Oct 18, 2024

jonathanpeppers commented Oct 18, 2024

Android API 23, Runtime Crash in .NET 9 #106025

Android API 23, Runtime Crash in .NET 9 #106025

Comments

jonathanpeppers commented Aug 6, 2024

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

dotnet-policy-service bot commented Aug 6, 2024

akoeplinger commented Aug 6, 2024

dotnet-policy-service bot commented Aug 9, 2024

dotnet-policy-service bot commented Aug 9, 2024

steveisok commented Aug 9, 2024

jonathanpeppers commented Aug 9, 2024

grendello commented Aug 19, 2024

akoeplinger commented Aug 19, 2024

akoeplinger commented Aug 19, 2024 • edited Loading

grendello commented Aug 19, 2024 • edited Loading

BrzVlad commented Sep 17, 2024 • edited Loading

jonathanpeppers commented Sep 17, 2024

jonathanpeppers commented Sep 17, 2024

jonathanpeppers commented Sep 17, 2024

jeffhandley commented Oct 8, 2024

jonathanpeppers commented Oct 14, 2024

jeffhandley commented Oct 16, 2024

tannergooding commented Oct 16, 2024

jonathanpeppers commented Oct 16, 2024

steveisok commented Oct 17, 2024

jonathanpeppers commented Oct 17, 2024 • edited Loading

matouskozak commented Oct 17, 2024

jonathanpeppers commented Oct 17, 2024

matouskozak commented Oct 18, 2024

jonathanpeppers commented Oct 18, 2024

akoeplinger commented Aug 19, 2024 •

edited

Loading

grendello commented Aug 19, 2024 •

edited

Loading

BrzVlad commented Sep 17, 2024 •

edited

Loading

jonathanpeppers commented Oct 17, 2024 •

edited

Loading