release/7.0 Android Mono runs are failing

[LoopedBard3]

Since 11/7,
Last good run: https://dev.azure.com/dnceng/internal/_build/results?buildId=2309910&view=results
First bad run: https://dev.azure.com/dnceng/internal/_build/results?buildId=2312318&view=logs&j=b3f9a873-839e-5600-b5cf-2f11099e60eb&t=a7e6f6ea-04c0-5565-1682-5f6c6d2084e5

[2023/11/23 01:41:42][INFO] [01:41:42] dbug: Executing command: 'D:\h\w\C69E0A35\p\microsoft.dotnet.xharness.cli\1.0.0-prerelease.21566.2\runtimes\any\native\adb\windows\adb.exe -s 09241JEC225722 install "D:\h\w\C69E0A35\p\performance\src\scenarios\helloandroid\pub\HelloAndroid.apk"'
[2023/11/23 01:43:00][INFO] [01:43:00] fail: Error:
[2023/11/23 01:43:00][INFO]                  Exit code: 1
[2023/11/23 01:43:00][INFO]                  Standard Output:
[2023/11/23 01:43:00][INFO]                  Performing Streamed Install
[2023/11/23 01:43:00][INFO] 
[2023/11/23 01:43:00][INFO]                  Standard Error:
[2023/11/23 01:43:00][INFO]                  adb: failed to install D:\h\w\C69E0A35\p\performance\src\scenarios\helloandroid\pub\HelloAndroid.apk: Failure [INSTALL_FAILED_VERIFICATION_FAILURE]

[LoopedBard3]

Searching for the [INSTALL_FAILED_VERIFICATION_FAILURE] failure, I found this https://stackoverflow.com/questions/15014519/apk-installation-failed-install-failed-verification-failure. Testing locally, it appears that the version the apps are targetting is old enough that the phones are popping up an error.

[fanyang-mono]

the version the apps are targetting is old enough that the phones are popping up an error.
By version, do you mean Android API version or something else?

[LoopedBard3]

The exact message in the pop-up was:

This app was built for an older version of Android and doesn't include the latest privacy protections. 

Installing this app may put your device at risk. Learn more (https://support.google.com/android/answer/2812853?hl=en) about Play Protect.

I included the link to the learn more inline. My understanding is that this is likely related to the target Android OS version specified by the app, with the default one for the release/7.0 android app being old enough to cause an error (https://developer.android.com/about/versions/10/behavior-changes-all#low-target-sdk-warnings). If this version can be updated for the sample apps in release/7.0, we should stop hitting this problem, but I am not familiar with how this would be done.

[fanyang-mono]

The app (https://github.com/dotnet/runtime/tree/release/7.0/src/mono/sample/Android) was configured to be built for API level 33. When testing locally, I hit this error when trying to install the app on a device with API level 34. I need to figure out how to change the target API level when building this app.

[fanyang-mono]

Probably needs to pass TargetApiLevel=34 to ApkBuilder for the Android sample app.

[LoopedBard3]

That makes sense, do you know correct path for passing the TargetApiLevel to the ApkBuilder? Although, looking at the ApkBuilder for release/7.0, it does not appear to have the option for TargetApiLevel: https://github.com/dotnet/runtime/blob/release/7.0/src/tasks/AndroidAppBuilder/ApkBuilder.cs#L15-L16 compared to main: https://github.com/dotnet/runtime/blob/main/src/tasks/AndroidAppBuilder/ApkBuilder.cs#L17, so whoever manages the ApkBuilder probably has to backport or add the TargetApiLevel change.

[fanyang-mono]

dotnet/runtime#95770 this should fix the problem. When it gets approved and released, the perf lab should switch to use the corresponding .NET7 servicing package. Then the app will be able to install successfully.

[fanyang-mono]

dotnet/runtime#95770 has been merged. Does the perf lab run .NET7 Android sample from dotnet/runtime release/7.0 branch?

[LoopedBard3]

Yes, we run and build directly from the respective release branches. In this case, we also seem to test some of the staging branch changes with this change being tested in: https://dev.azure.com/dnceng/internal/_build/results?buildId=2333516&view=results. Although the results should not show up in the reports as we only take release/*.0 and main for the reports.

[fanyang-mono]

Make sense. Will you be able to test the change from staging branch on perf lab device to make sure the issue is gone? Then, we probably could close this issue.

[LoopedBard3]

Looks like the app build failed because the BuildApiLevel (29) was less than the TargetApiLevel (31). https://dev.azure.com/dnceng/internal/_build/results?buildId=2333516&view=results

../../../../dotnet.sh publish \
/p:TargetArchitecture=arm64 \
/p:TargetOS=Android \
/p:Configuration=Release \
/p:DeployAndRun=false \
/p:RunAOTCompilation=false \
/p:ForceAOT=false \
/p:AOTWithLibraryFiles=false \
/p:MonoForceInterpreter=false \
/p:UseLLVM=true \
/p:RunActivity=false \
'/p:RuntimeComponents=""' \
'/p:DiagnosticPorts=""'
MSBuild version 17.4.8+6918b863a for .NET
  Determining projects to restore...
  Restored /__w/1/s/src/mono/sample/Android/AndroidSampleApp.csproj (in 808 ms).
  Used runtime pack: /__w/1/s/artifacts/bin/microsoft.netcore.app.runtime.android-arm64/Release/
  AndroidSampleApp -> /__w/1/s/artifacts/bin/AndroidSampleApp/arm64/Release/android-arm64/AndroidSampleApp.dll
  Optimizing assemblies for size. This process might take a while.
  Path: /__w/1/s/artifacts/bin/AndroidSampleApp/arm64/Release/android-arm64/publish/
  SourceDir: /__w/1/s/artifacts/bin/AndroidSampleApp/arm64/Release/android-arm64/
/__w/1/s/src/mono/sample/Android/AndroidSampleApp.csproj(81,5): error MSB4018: The "AndroidAppBuilderTask" task failed unexpectedly.
/__w/1/s/src/mono/sample/Android/AndroidSampleApp.csproj(81,5): error MSB4018: System.ArgumentException: BuildApiLevel=29 < TargetApiLevel=31. Make sure you've downloaded some recent build-tools in Android SDK
/__w/1/s/src/mono/sample/Android/AndroidSampleApp.csproj(81,5): error MSB4018:    at ApkBuilder.BuildApk(String abi, String mainLibraryFileName, String monoRuntimeHeaders) in /_/src/tasks/AndroidAppBuilder/ApkBuilder.cs:line 145
/__w/1/s/src/mono/sample/Android/AndroidSampleApp.csproj(81,5): error MSB4018:    at AndroidAppBuilderTask.Execute() in /_/src/tasks/AndroidAppBuilder/AndroidAppBuilder.cs:line 128
/__w/1/s/src/mono/sample/Android/AndroidSampleApp.csproj(81,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
/__w/1/s/src/mono/sample/Android/AndroidSampleApp.csproj(81,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)
Makefile:25: recipe for target 'run' failed

I am not sure what/where would need to change to fix the exception. If you get a PR or branch with a change you want to test, let me know and I can run just the test for the Android app rather than having to make a PR, merge, and then wait for a test.

[fanyang-mono]

This is because the Android SDK version installed on your computer is API level 29. You need to follow the script in this doc to install newer Android SDK.

[LoopedBard3]

Do you happen to know who has done this for the runtime repo in the past? This failure is from within the azure pipelines so it is using a container. I thought I had saw a commit mentioning updating it but haven't been able to find it again.

[LoopedBard3]

It looks like this is the commit I was thinking of: dotnet/runtime@921029b, though that was a decent time ago. I also found that it looks like the latest images were set to be used on main in 2022: https://github.com/dotnet/runtime/pull/76630/files, maybe doing something similar would work.

[fanyang-mono]

Let me know if that works. Note that .NET8 has the latest change. Use the same version of the container as being used to build the Android app for .NET8 would probably fix this issue.

[LoopedBard3]

Fixed by: dotnet/runtime@d776e5e