shared_camera_java sample crashes on some devices if camera is reconfigured

[rickyhopper]

SPECIFIC ISSUE ENCOUNTERED

When setting up an ARCore session with Session.Feature.SHARED_CAMERA, setting the CameraConfig to certain configs provided by Session.getSupportedCameraConfigs(CameraConfigFilter) causes ARCore to crash on some devices. I have reproduced this issue by adding a few lines of code to the shared_camera_java example in this repository, as detailed in the repro steps.

VERSIONS USED

• Android Studio: 2023.3.1 Patch 1, Build #AI-233.14808.21.2331.11842104
• ARCore SDK for Android: 1.43.0 (in sample) and 1.44.0 (in my app)
• Device manufacturer, model, and O/S: Google Pixel 3 (Android 12, Build SP1A.210812.016.C1)
• Google Play Services for AR (ARCore): 1.44.241490493
  On Windows, use: adb shell pm dump com.google.ar.core | findstr /i "packages: versionName"
  On macOS, use: adb shell pm dump com.google.ar.core | egrep -i versionName\|packages:
• Output of adb shell getprop ro.build.fingerprint: google/blueline/blueline:12/SP1A.210812.016.C1/8029091:user/release-keys

STEPS TO REPRODUCE THE ISSUE

1. Set a high resolution camera config when configuring your session. The below is example code I inserted in shared_camera_java to sanity test this (in SharedCameraActivity::openCamera at line 598):

      errorCreatingSession = false;

      // NEW CODE STARTS HERE
      CameraConfigFilter filter = new CameraConfigFilter(sharedSession);
      filter.setFacingDirection(CameraConfig.FacingDirection.BACK);
      List<CameraConfig> ccList = sharedSession.getSupportedCameraConfigs(filter);
      for (CameraConfig cc : ccList) {
        Size imgSize = cc.getImageSize();
        if (imgSize.getWidth() > 1000 && imgSize.getHeight() > 1000) {
          // this is the first config which is high enough resolution for my use case
          sharedSession.setCameraConfig(cc);
          break;
        }
      }
      // NEW CODE ENDS HERE

      // Enable auto focus mode while ARCore is running.
      Config config = sharedSession.getConfig();
      config.setFocusMode(Config.FocusMode.AUTO);
      sharedSession.configure(config);

2. Run the app on an older Android device (Pixel 3 details given above, also happens on a Galaxy Note 9)
3. Notice the app crashes with the following issue:

Session 0: Failed to create capture session; configuration failed
CameraAccessException
android.hardware.camera2.CameraAccessException: CAMERA_ERROR (3): endConfigure:559: Camera 0: Error configuring streams: Function not implemented (-38)
	at android.hardware.camera2.CameraManager.throwAsPublicException(CameraManager.java:1179)
	at android.hardware.camera2.impl.ICameraDeviceUserWrapper.endConfigure(ICameraDeviceUserWrapper.java:119)
	at android.hardware.camera2.impl.CameraDeviceImpl.configureStreamsChecked(CameraDeviceImpl.java:508)
	at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSessionInternal(CameraDeviceImpl.java:715)
	at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSession(CameraDeviceImpl.java:552)
	at com.google.ar.core.examples.java.sharedcamera.SharedCameraActivity.createCameraPreviewSession(SharedCameraActivity.java:539)
	at com.google.ar.core.examples.java.sharedcamera.SharedCameraActivity.access$200(SharedCameraActivity.java:102)
	at com.google.ar.core.examples.java.sharedcamera.SharedCameraActivity$1.onOpened(SharedCameraActivity.java:222)
	at com.google.ar.core.al.run(Unknown Source:15)
	at android.os.Handler.handleCallback(Handler.java:938)
	at android.os.Handler.dispatchMessage(Handler.java:99)
	at android.os.Looper.loopOnce(Looper.java:201)
	at android.os.Looper.loop(Looper.java:288)
	at android.os.HandlerThread.run(HandlerThread.java:67)
Caused by: android.os.ServiceSpecificException: endConfigure:559: Camera 0: Error configuring streams: Function not implemented (-38) (code 10)
	at android.os.Parcel.createExceptionOrNull(Parcel.java:2439)
	at android.os.Parcel.createException(Parcel.java:2409)
	at android.os.Parcel.readException(Parcel.java:2392)
	at android.os.Parcel.readException(Parcel.java:2334)
	at android.hardware.camera2.ICameraDeviceUser$Stub$Proxy.endConfigure(ICameraDeviceUser.java:823)
	at android.hardware.camera2.impl.ICameraDeviceUserWrapper.endConfigure(ICameraDeviceUserWrapper.java:116)
	at android.hardware.camera2.impl.CameraDeviceImpl.configureStreamsChecked(CameraDeviceImpl.java:508) 
	at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSessionInternal(CameraDeviceImpl.java:715) 
	at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSession(CameraDeviceImpl.java:552) 
	at com.google.ar.core.examples.java.sharedcamera.SharedCameraActivity.createCameraPreviewSession(SharedCameraActivity.java:539) 
	at com.google.ar.core.examples.java.sharedcamera.SharedCameraActivity.access$200(SharedCameraActivity.java:102) 
	at com.google.ar.core.examples.java.sharedcamera.SharedCameraActivity$1.onOpened(SharedCameraActivity.java:222) 
	at com.google.ar.core.al.run(Unknown Source:15) 
	at android.os.Handler.handleCallback(Handler.java:938) 
	at android.os.Handler.dispatchMessage(Handler.java:99) 
	at android.os.Looper.loopOnce(Looper.java:201) 
	at android.os.Looper.loop(Looper.java:288) 
	at android.os.HandlerThread.run(HandlerThread.java:67) 

WORKAROUNDS (IF ANY)

None.

ADDITIONAL COMMENTS

I am trying to use the highest resolution camera config possible to capture frames for a 3D reconstruction application. I cannot determine which configs are supported by the cameras on my test devices because most CameraConfigs crash (both the high-resolution one gotten with the code above and the medium resolution one provided in the list fail to configure). There is nothing in the documentation about any other way to determine if a CameraConfig is supported, and as far as I can tell I am doing what I am supposed to by calling session.configure after setting the CameraConfig.

Thanks for any assistance on this; and I am happy to help dig into this however I can.

[rickyhopper]

Update on this. I continued to dig into it, and the actual issue happens when the ImageReader surface for CPU processing is added to the capture request, and then the capture request is created. The camera on Pixel 3 and other devices can't seem to handle this, and it's unclear why. My working theory is that it's just streaming to too many surfaces (since there is the GPU surface, the VGA motion tracking surface, the HD surface, and the ImageReader surface all provided), but I can't find anything to back this up. It seems like this still shouldn't crash, and lies somewhere between ARCore and the camera2 library.

I've updated the title to reflect this, but leaving this bug open since it relates to the shared_camera_java sample.