Hotwire Native Camera Access

In this blog post we will look at how to access the camera and show a camera feed in Hotwire Native.

Before we start, if you are only interested in capturing a photo or selecting a photo from the gallery, good news: It's already built-in into Hotwire Native 🎉
You can just use an input field with the type file like this:

<input type="file" accept="image/*">

<!-- Or go directly to the "Take Photo" mode -->
<input type="file" accept="image/*" capture="camera">

Note: It works out of the box for Android. But for iOS you need to add a usage description to your Info.plist for NSCameraUsageDescription (Privacy - Camera Usage Description), where you explain why the app needs camera access.

In this article however we will focus on accessing the camera and show an inline camera feed in the app.

This can be useful for things like:

QR-Code scanning
Document scanning
Any sort of object detection

Web

The most basic snippet to access the camera feed in the browser looks like this:

<button id="camera-access">Open Camera</button>
<video></video>

document.querySelector("#camera-access").addEventListener("click", () => {
  const constraints = {
    audio: false,
    video: true,
  };
  navigator.mediaDevices
    .getUserMedia(constraints)
    .then((mediaStream) => {
      const video = document.querySelector("video");
      video.srcObject = mediaStream;
      video.onloadedmetadata = () => {
        video.play();
      };
    })
});

Two gotchas when working with getUserMedia:

You need to have a onloadedmetadata callback, otherwise you might see the camera access notification light flip to green to show the camera is enabled, but you might not be able to see the feed at all
Camera access is only allowed on HTTPS urls. If you are using a HTTP url, you will get a NotAllowedError: Permission denied error

There are many ways to set up local HTTPS. Personally, I like using ngrok since I'm very lazy and it's super easy to use.

Before we dive in further, it will be quite handy later on in this article to have the above code in a Stimulus controller.
The code we work with in the following will look like this:

import { Controller } from "@hotwired/stimulus"

// Connects to data-controller="camera"
export default class extends Controller {
  static targets = ["video"]

  startCamera() {
    navigator.mediaDevices.getUserMedia(this.videoOptions).then((mediaStream) => {
      this.videoTarget.srcObject = mediaStream
      this.videoTarget.onloadedmetadata = () => {
        this.videoTarget.play()
      }
    })
  }

  get videoOptions() {
    return {
      audio: false,
      video: true,
      // Note: You can also specify the camera you want to use
      // video: { facingMode: "environment" }
    }
  }
}

This controller can be used like this:

<div data-controller="camera">
  <button data-action="click->camera#startCamera">Open Camera</button>
  <video data-camera-target="video"></video>
</div>

Now that we have the web implementation set up, let’s move on to the native integrations.

iOS

First, you need to add usage descriptions to your Info.plist file like mentioned above.
For the camera, you need to add NSCameraUsageDescription and explain why you need camera access.

Next you need to allow the allowsInlineMediaPlayback property in the WebView configuration. Otherwise you won't see the camera feed.
Wherever you configure your Hotwire settings:

Hotwire.config.makeCustomWebView = { configuration in
    configuration.allowsInlineMediaPlayback = true
    return WKWebView(frame: .zero, configuration: configuration)
}

If you try to access the camera now (through an HTTPS url), iOS will ask you for camera access, and after granting it, you should see the camera feed in your app.
There is only one catch: After closing and reopening the app, you'll be asked for camera access again.
We need to explicitly set a custom WKUIDelegate that automatically grants media capture permissions, so the user isn’t prompted every time the app is reopened—after they have granted permission initially.

@preconcurrency import WebKit
import HotwireNative

class CustomWKUIController: WKUIController {
    func webView(
        _ webView: WKWebView,
        requestMediaCapturePermissionFor origin: WKSecurityOrigin,
        initiatedByFrame frame: WKFrameInfo,
        type: WKMediaCaptureType,
        decisionHandler: @escaping (WKPermissionDecision) -> Void) {
            decisionHandler(.grant)
        }
}

Note: We subclass Hotwire Native's WKUIController here to avoid losing the built-in alert() and prompt() functionality that Hotwire provides. Thanks to Joe Masilotti for pointing this out!

HotwireNative has a public function to set our custom WKUIDelegate.
So wherever we configure our Navigator, we can then set our custom WKUIDelegate like this:

private init() {
    configureHotwire()
    self.navigator = Navigator()
+   self.navigator.webkitUIDelegate = CustomWKUIController(delegate: navigator)
}

Done 🎉

Very little code is needed to get the camera feed working in iOS.
The Android part takes a bit more code.

Android

Let’s start by telling Android we wanna have access to camera at some point.
This is done by adding the following two lines to the AndroidManifest.xml file:

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" android:required="false" />

Next we need to allow permission request coming from the WebView. In contrast to iOS, the WebView doesn't propagate the permission request to the app.
We need to handle that manually.
Luckily, it's pretty easy to do so. We just need to create a custom WebChromeClient and override the onPermissionRequest method.

Let's create a custom Chrome client that always grants the camera permission:

class CustomChromeClient(session: Session): HotwireWebChromeClient(session) {
    override fun onPermissionRequest(request: PermissionRequest?) {
        // If permission request is for camera access
        if (request?.resources?.contains(PermissionRequest.RESOURCE_VIDEO_CAPTURE) == true) {
            // Always grant permission
            // Depending on the android camera permission, the browser will still get a permission denied error if the user has not granted permission
            request.grant(request.resources)
        }
    }
}

To use this Chrome client, we can simply override the createWebChromeClient method in our WebFragment:

open class WebFragment : HotwireWebFragment() {
    override fun createWebChromeClient(): HotwireWebChromeClient {
        return CustomChromeClient(navigator.session)
    }
}

You might be wondering why we always grant the camera permission and whether this is a safe thing to do.
Well, when you test the above code, you'll see that the camera feed still won't show up even though we granted the camera permission.
Only thing you see is a permission denied error in the JS console.

Here comes the tricky part of camera access on Android:
In a perfect world, we would ask the user for the camera permission inside the onPermissionRequest method and respond the user's decision to Chrome.

But by the time the onPermissionRequest method is called, it’s already too late to ask the user for camera permission. As far as I know, there is no way to request camera permission, wait for the user’s response, and then grant or deny the permission request.
This is due to the async nature of Android and that we get the permission response at a completely different place in the code. You can learn more about Android runtime permissions here.

There are multiple solutions for that. All of them involve asking the user before the onPermissionRequest method gets called.
You could for example ask for camera permission in the MainActivity, right after the app starts.
That way, when Chrome asks for the camera permission, the user has either already granted it and everything works, or if they denied it, you'll get the JS permission denied error.

This approach might be fine for some apps, but I don't like it very much.

The reason is that we request permissions without the user knowing why we need them. They never interacted with the camera button, yet they are still asked for access
This isn’t as user-friendly as how iOS handles it; only after the user interacts with the camera button do they get asked for camera permission.

One other thing I've done in the past, is asking for the camera permission when the WebView that includes the camera feature gets loaded. (Via a small Bridge component)
It looked something like this:

<div data-controller="native--bridge" data-native--bridge-action-value="askForPermission" data-native--bridge-payload-value="<%= { permission: 'camera' }.to_json %>">
  <div data-controller="qr-scanner">
    <button data-action="click->qr-scanner#start">
      Start QR-Scanner
    </button>
  </div>
</div>

That way, the user at least doesn't get ask right after the app starts. He will be asked when he navigates to a page that needs the camera.
Still not the best user experience. Maybe the camera feature is only a small part of the page.

The goal should be to ask for the camera permission when the user interacts with the camera button.

To accomplish this, we can create a bridge component called “permissions”. When the user interacts with the camera button, we ask for the camera permission.
After we received the user decision we either start the actual camera feed through getUserMedia or show some sort of permission denied message.

The JS part of the bridge component looks like this:

import { BridgeComponent } from "@hotwired/hotwire-native-bridge"

const PERMISSIONS = {
  CAMERA: "CAMERA",
}

// Connects to data-controller="native--permissions"
export default class extends BridgeComponent {
  static component = "permissions"

  checkPermissions({ params: { permission } }) {
    const sanitizedPermission = PERMISSIONS[permission]
    if (!sanitizedPermission) {
      console.warn(`Unknown permission: ${permission}`)
      return
    }

    this.send("checkPermissions", { permission: sanitizedPermission }, (message) => {
      const granted = message.data.granted

      // Dispatches a "result" event with the granted status
      this.dispatch("result", { granted: granted })

      // If an action only needs to be taken if the permission is granted,
      // we can directly listen for the "granted" or "denied" events
      if (granted) {
        this.dispatch("granted")
      } else {
        this.dispatch("denied")
      }
    })
  }
}

The native counterpart of the Bridge component:

class PermissionsComponent(
    name: String,
    private val hotwireDelegate: BridgeDelegate<HotwireDestination>
) : BridgeComponent<HotwireDestination>(name, hotwireDelegate) {

    private val fragment: Fragment
        get() = hotwireDelegate.destination.fragment

    override fun onReceive(message: Message) {
        when (message.event) {
            "checkPermissions" -> checkPermissions(message)
            else -> Log.w("PermissionsComponent", "Unknown event for message: $message")
        }
    }

    private fun checkPermissions(message: Message) {
        val data = message.data<MessageData>() ?: return

        val permission = Permissions.fromString(data.permission) ?: run {
            Log.w("PermissionsComponent", "Unknown permission: ${data.permission}")
            return
        }

        if (hasPermission(permission.permissionString)) {
            replyTo("checkPermissions", PermissionResultData(true))
        } else {
            (fragment as? PermissionRequester)?.requestPermission(permission.permissionString) { isGranted ->
                replyTo("checkPermissions", PermissionResultData(isGranted))
            } ?: Log.w("PermissionsComponent", "Fragment does not implement PermissionRequester")
        }
    }

    private fun hasPermission(permission: String): Boolean {
        return ContextCompat.checkSelfPermission(
            fragment.requireContext(),
            permission
        ) == PackageManager.PERMISSION_GRANTED
    }

    @Serializable
    data class MessageData(
        @SerialName("permission") val permission: String
    )

    @Serializable
    data class PermissionResultData(
        @SerialName("granted") val granted: Boolean
    )

    sealed class Permissions(val permissionString: String) {
        data object Camera : Permissions(Manifest.permission.CAMERA)

        companion object {
            fun fromString(permission: String): Permissions? {
                return when (permission) {
                    "CAMERA" -> Camera
                    else -> null
                }
            }
        }
    }
}

This Permission component can't ask for permissions directly. It needs some help from either an activity or fragment.
In the code above you'll see that is uses a PermissionRequester interface that is expected to be implemented by the fragment that includes the WebView.

The Interface can be very simple:

interface PermissionRequester {
    fun requestPermission(permission: String, callback: (Boolean) -> Unit)
}

Now we only need to implement the requestPermission method in our WebFragment:

open class WebFragment : HotwireWebFragment(), PermissionRequester {
    private var permissionCallback: ((Boolean) -> Unit)? = null

    private val requestPermissionLauncher = registerForActivityResult(
        ActivityResultContracts.RequestPermission()
    ) { isGranted: Boolean ->
        permissionCallback?.invoke(isGranted)
        permissionCallback = null
    }

    override fun requestPermission(permission: String, callback: (Boolean) -> Unit) {
        permissionCallback = callback
        requestPermissionLauncher.launch(permission)
    }
    ...
}

That's quite a bit of Android code. But the general idea is quite simple. The bridge component reaches out to the Fragment to ask for the camera permission. The Fragment asks the user for the permission and informs the bridge component about the user's decision.
This user decision then gets sent back to the WebView.

In the web we can now listen for the granted or denied event and act accordingly.

We would use it in the HTML like this:

<div data-controller="native--permissions camera" data-action="native--permissions:granted->camera#startCamera">
  <button data-action="click->native--permissions#checkPermissions" data-native--permissions-permission-param="CAMERA">Open Camera</button>
  <video data-camera-target="video"></video>
</div>

That's it. Now you can ask for permissions on the Android level and inform the WebView about the current decision.
It's build in a way, that you can easily extend it with more permissions. Just add a new permission to the Permissions sealed class and ask for it in the web.
It's more complex than the iOS solution, but at least the user experience is the same.

But there is one catch…

iOS (again)

We now have two different ways of starting the camera feed. The iOS that handles the camera permission request so we can start the camera feed directly, and the Android way that asks for the camera permission before we can start the camera feed.

There are two basic ways to handle this. We could either handle the difference in the web like this:

<div data-controller="native--permissions camera" data-action="native--permissions:granted->camera#startCamera">
  <% action = ios_app? ? "camera#startCamera" : "native--permissions#checkPermissions" %>
  <button data-action="<%= action %>" data-native--permissions-permission-param="CAMERA">Open Camera</button>
  <video data-camera-target="video"></video>
</div>

Or we can create a similar permission component in iOS, so our web code stays the same.

The iOS permission component could look like this:

import Foundation
import HotwireNative
import AVFoundation

final class PermissionsComponent: BridgeComponent {
    override class var name: String { "permissions" }

    override func onReceive(message: Message) {
        guard let event = Event(rawValue: message.event) else {
            return
        }

        switch event {
        case .checkPermissions:
            handleCheckPermissions(message: message)
        }
    }

    private func handleCheckPermissions(message: Message) {
        guard let data: MessageData = message.data() else { return }
        let permission = Permissions.fromString(data.permission)

        switch permission {
        case .camera:
            handleCameraPermission(message: message)
        case .unknown:
            print("PermissionsComponent - Unknown permission: \(permission)")
        }
    }

    private func handleCameraPermission(message: Message) {
        let status = AVCaptureDevice.authorizationStatus(for: .video)

        switch status {
        case .authorized:
            reply(to: Event.checkPermissions.rawValue, with: PermissionResultData(granted: true))
        case .denied, .restricted:
            reply(to: Event.checkPermissions.rawValue, with: PermissionResultData(granted: false))
        case .notDetermined:
            // Camera access has not been requested yet
            AVCaptureDevice.requestAccess(for: .video) { granted in
                DispatchQueue.main.async {
                    self.reply(to: Event.checkPermissions.rawValue, with: PermissionResultData(granted: granted))
                }
            }
        @unknown default:
            reply(to: Event.checkPermissions.rawValue, with: PermissionResultData(granted: false))
        }
    }
}

// MARK: Events

private extension PermissionsComponent {
    enum Event: String {
        case checkPermissions
    }
}

// MARK: Message data

private extension PermissionsComponent {
    struct MessageData: Decodable {
        let permission: String
    }

    struct PermissionResultData: Encodable {
        let granted: Bool
    }

    enum Permissions {
        case camera
        case unknown

        static func fromString(_ permission: String) -> Permissions {
            switch permission.lowercased() {
            case "camera":
                return .camera
            default:
                return .unknown
            }
        }
    }
}

Done. Now we have a similar permission component in iOS and Android and can handle the camera permission request in the same way in the web.

In this post, we covered how to access the camera feed in Hotwire Native, both in the web and native environments.
Ensuring a smooth user experience on both iOS and Android.

If you have any questions or run into problems, don’t hesitate to reach out!

And you can find the full code in this Pull Request.

Camera Access With Hotwire Native

Comments

More from this blog

Hotwire Native - Switch Environments

Web

iOS

Android

iOS (again)

Command Palette

Comments

More from this blog

Web

iOS

Android

iOS (again)