mobile-camera-vision-camera
VisionCamera v4+ - photo/video capture, QR/barcode scanning, real-time frame processors, zoom/focus/exposure, HDR, location metadata, format selection
Install
npx skills add https://github.com/agents-inc/skills/tree/main/dist/plugins/mobile-camera-vision-camera/skills/mobile-camera-vision-camera
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install agents-inc-skills@llmmart
git clone https://github.com/agents-inc/skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole agents-inc/skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
VisionCamera Patterns
Quick Guide: Use VisionCamera for high-performance camera features in React Native. Control the camera lifecycle with
isActive(never unmount/remount). UseuseCameraDeviceto select back/front cameras,useCameraPermissionfor permissions. Capture photos withtakePhoto(), record video withstartRecording()/stopRecording(), scan codes withuseCodeScanner, and process frames in real time withuseFrameProcessorworklets. Frame processors run on a parallel JS thread via JSI -- keep them fast or userunAsync/runAtTargetFpsto avoid blocking the pipeline.
<critical_requirements>
CRITICAL: Before Using This Skill
All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering,
import type, named constants)
(You MUST set isActive based on screen focus AND app state -- camera must pause when backgrounded or navigated away)
(You MUST request permissions before rendering the Camera -- useCameraPermission returns hasPermission and requestPermission)
(You MUST include the 'worklet' directive as the first line of every frame processor function body)
(You MUST enable only the pipelines you need (photo, video, codeScanner, frameProcessor) -- unused pipelines waste resources)
(You MUST use useSharedValue (not useState) for data shared between frame processors and the React thread)
</critical_requirements>
Auto-detection: VisionCamera, react-native-vision-camera, useCameraDevice, useCameraDevices, useCameraPermission, useMicrophonePermission, useCodeScanner, useFrameProcessor, useSkiaFrameProcessor, useCameraFormat, takePhoto, takeSnapshot, startRecording, stopRecording, Camera component, frame processor, worklet, codeScanner, photoQualityBalance, enableLocation, videoHdr, photoHdr
When to use:
- Capturing photos or recording video in a React Native app
- Scanning QR codes or barcodes (EAN-13, Code-128, etc.)
- Real-time frame processing for ML, object detection, or image analysis
- Implementing zoom, focus, exposure, or HDR controls
- Embedding GPS location metadata in captured media
- Selecting specific camera devices (ultra-wide, telephoto, front/back)
When NOT to use:
- Picking images from the device gallery (use an image picker)
- Simple static image display (use standard Image component)
- Web-only camera access (use browser MediaDevices API)
Key patterns covered:
- Camera lifecycle management with
isActiveand screen/app state - Permission handling with hooks (
useCameraPermission,useMicrophonePermission) - Photo capture (
takePhoto,takeSnapshot) and video recording - QR/barcode scanning with
useCodeScanner - Frame processors with worklets,
runAsync, andrunAtTargetFps - Device selection, format selection, zoom, focus, exposure, HDR
- Location metadata embedding
- Performance optimization (pipeline selection, buffer compression, pixel format)
Detailed Resources:
- examples/core.md - Camera setup, permissions, lifecycle, photo capture, video recording
- examples/scanning-and-processing.md - Code scanning, frame processors, worklet patterns
- reference.md - Decision frameworks, device/format selection, performance checklist
<decision_framework>
Decision Framework
Capture Method
What do you need to capture?
|
+-> Still image?
| +-> High quality (AE/AF/AWB) → takePhoto()
| +-> Fast preview capture (~16ms) → takeSnapshot() (requires video pipeline)
|
+-> Video?
| +-> Continuous recording → startRecording() / stopRecording()
| +-> Need pause/resume → pauseRecording() / resumeRecording()
| +-> User cancelled → cancelRecording()
|
+-> QR/barcode scanning?
| +-> useCodeScanner (native thread, no frame processor needed)
|
+-> Real-time frame analysis (ML, detection)?
+-> useFrameProcessor with native plugins
+-> Need Skia drawing on frames? → useSkiaFrameProcessor
Frame Processor Scheduling
How fast must your processor run?
|
+-> Every frame (30/60 FPS)?
| +-> Processing < 33ms? → Default synchronous processor
| +-> Processing > 33ms? → runAsync (offload to separate thread)
|
+-> Lower rate is fine (5-10 FPS)?
+-> runAtTargetFps(targetFps, () => { ... })
Device Selection
Which camera?
|
+-> Simple back/front → useCameraDevice("back") or useCameraDevice("front")
+-> Multi-lens (0.5x + 1x + 3x) → useCameraDevice("back", { physicalDevices: [...] })
+-> External USB camera → Filter useCameraDevices() for position === "external"
+-> Custom logic → useCameraDevices() + useMemo with your own filter
</decision_framework>
<red_flags>
RED FLAGS
High Priority Issues:
- Mounting/unmounting Camera instead of toggling
isActive-- wastes resources, slow resume - Missing
'worklet'directive in frame processor function -- silently runs on wrong thread, crashes - Using
useStateto share data from frame processors -- causes thread context switching, useuseSharedValue - Enabling all pipelines (
photo,video,codeScanner, frame processor) when only one is needed -- wastes memory and battery - Not requesting permissions before rendering Camera -- causes crash or blank preview
- Calling
camera.current.takePhoto()beforeonInitializedfires -- method not ready, throws error
Medium Priority Issues:
- Capturing 4K when 1080p is sufficient -- wastes memory, slower processing
- Not debouncing
onCodeScanned-- fires many times per second, causes excessive state updates - Using
useSkiaFrameProcessorwhenuseFrameProcessorsuffices -- Skia adds overhead - Enabling
videoHdrwithout checkingformat.supportsVideoHdr-- crashes on unsupported formats - Not checking
device.supportsFocusbefore callingfocus()-- fails on devices without AF
Gotchas & Edge Cases:
- Zoom is logarithmic -- 1x to 2x is a much bigger visual change than 127x to 128x. Use
interpolate()for linear gesture mapping. takeSnapshot()requiresvideo={true}on iOS -- it captures from the video preview buffer- Frame processors at 4K process ~12MB per frame -- use lower resolution if possible
device.neutralZoommay not be 1.0 on ultra-wide cameras -- always use it as the starting zoom- Android code scanner requires MLKit (
VisionCamera_enableCodeScanner=true) -- without it, scanning silently does nothing - UPC-A codes report as EAN-13 on iOS (EAN-13 is a superset)
cancelRecording()firesonRecordingErrorwithcapture/recording-canceled-- handle this error type gracefullyenableLocationrequires platform manifest entries (iOS:NSLocationWhenInUseUsageDescription, Android:ACCESS_FINE_LOCATION)- Disable location APIs entirely via build flag if unused -- prevents App Store rejection for unnecessary privacy APIs
exposureprop is an offset from auto-exposure, not an absolute ISO value- Video HDR uses 10-bit pixel format which adds processing overhead -- disable when not needed
</red_flags>
<critical_reminders>
CRITICAL REMINDERS
All code must follow project conventions in CLAUDE.md
(You MUST set isActive based on screen focus AND app state -- camera must pause when backgrounded or navigated away)
(You MUST request permissions before rendering the Camera -- useCameraPermission returns hasPermission and requestPermission)
(You MUST include the 'worklet' directive as the first line of every frame processor function body)
(You MUST enable only the pipelines you need (photo, video, codeScanner, frameProcessor) -- unused pipelines waste resources)
(You MUST use useSharedValue (not useState) for data shared between frame processors and the React thread)
Failure to follow these rules will cause crashes, blank previews, wasted battery, and dropped frames.
</critical_reminders>
Files (skills)
-
examples
-
core.md 10.5 KB
# VisionCamera - Core Patterns > Camera setup, permissions, lifecycle, photo capture, and video recording. See [SKILL.md](../SKILL.md) for decision guidance and red flags. **Prerequisites**: react-native-vision-camera v4+, react-native-worklets-core (for frame processors). --- ## Pattern 1: Camera Lifecycle with Permissions The camera should only be active when the screen is focused and the app is in the foreground. Always handle permissions before rendering the Camera. ```typescript import { useCallback, useRef, useState } from "react"; import { StyleSheet, View, Text, Pressable } from "react-native"; import { Camera, useCameraDevice, useCameraPermission, type PhotoFile, } from "react-native-vision-camera"; import { useIsFocused } from "@react-navigation/native"; import { useAppState } from "@react-native-community/hooks"; export function CameraScreen() { const camera = useRef<Camera>(null); const device = useCameraDevice("back"); const { hasPermission, requestPermission } = useCameraPermission(); // Camera active ONLY when screen focused AND app foregrounded const isFocused = useIsFocused(); const appState = useAppState(); const isActive = isFocused && appState === "active"; if (!hasPermission) { return ( <View style={styles.container}> <Text>Camera permission is required</Text> <Pressable onPress={requestPermission}> <Text>Grant Permission</Text> </Pressable> </View> ); } if (device == null) { return ( <View style={styles.container}> <Text>No camera device found</Text> </View> ); } return ( <Camera ref={camera} style={StyleSheet.absoluteFill} device={device} isActive={isActive} photo={true} onInitialized={() => console.log("Camera ready")} onError={(error) => console.error("Camera error:", error)} /> ); } const styles = StyleSheet.create({ container: { flex: 1, alignItems: "center", justifyContent: "center" }, }); ``` **Why good:** permission checked before Camera renders, device null-checked, isActive combines focus + app state, camera not unmounted when inactive (toggle isActive instead), only `photo` pipeline enabled ```typescript // Bad: unmounting camera when navigating away {isFocused && <Camera device={device} isActive={true} />} ``` **Why bad:** unmounting and remounting is much slower than toggling isActive, loses camera session warmth --- ## Pattern 2: Photo Capture with Options ```typescript import { useCallback, useRef, useState } from "react"; import { Alert, Pressable, StyleSheet, Text, View } from "react-native"; import { Camera, useCameraDevice, useCameraFormat, type PhotoFile, } from "react-native-vision-camera"; const PHOTO_WIDTH = 1920; const PHOTO_HEIGHT = 1080; export function PhotoCaptureScreen() { const camera = useRef<Camera>(null); const device = useCameraDevice("back"); const [isReady, setIsReady] = useState(false); const [lastPhoto, setLastPhoto] = useState<PhotoFile | null>(null); // Select format matching desired resolution const format = useCameraFormat(device, [ { photoResolution: { width: PHOTO_WIDTH, height: PHOTO_HEIGHT } }, ]); const handleTakePhoto = useCallback(async () => { if (!camera.current || !isReady) return; try { const photo = await camera.current.takePhoto({ flash: "auto", enableShutterSound: true, enableAutoRedEyeReduction: true, }); setLastPhoto(photo); // photo.path is the temporary file path // photo.width, photo.height for dimensions } catch (error) { Alert.alert("Capture failed", String(error)); } }, [isReady]); // Fast snapshot alternative (~16ms, lower quality) const handleTakeSnapshot = useCallback(async () => { if (!camera.current || !isReady) return; const SNAPSHOT_QUALITY = 85; try { const snapshot = await camera.current.takeSnapshot({ quality: SNAPSHOT_QUALITY, }); setLastPhoto(snapshot); } catch (error) { Alert.alert("Snapshot failed", String(error)); } }, [isReady]); if (device == null) return null; return ( <View style={styles.container}> <Camera ref={camera} style={StyleSheet.absoluteFill} device={device} isActive={true} photo={true} format={format} photoQualityBalance="balanced" onInitialized={() => setIsReady(true)} /> <Pressable onPress={handleTakePhoto} disabled={!isReady}> <Text>Take Photo</Text> </Pressable> </View> ); } const styles = StyleSheet.create({ container: { flex: 1 }, }); ``` **Why good:** format selected for target resolution, onInitialized guards against calling takePhoto before camera ready, error handled with try/catch, snapshot offered as fast alternative, named constant for snapshot quality **Key distinctions:** | Method | Speed | Quality | Requires | | ---------------- | ------------------ | ------------------- | --------------------- | | `takePhoto()` | Slower (AE/AF/AWB) | Full sensor quality | `photo={true}` | | `takeSnapshot()` | ~16ms | Preview quality | `video={true}` on iOS | --- ## Pattern 3: Video Recording with Pause/Resume ```typescript import { useCallback, useRef, useState } from "react"; import { Pressable, StyleSheet, Text, View } from "react-native"; import { Camera, useCameraDevice, useMicrophonePermission, type VideoFile, } from "react-native-vision-camera"; export function VideoRecordingScreen() { const camera = useRef<Camera>(null); const device = useCameraDevice("back"); const { hasPermission: hasMicPermission, requestPermission: requestMicPermission } = useMicrophonePermission(); const [isRecording, setIsRecording] = useState(false); const [isPaused, setIsPaused] = useState(false); const handleStartRecording = useCallback(() => { if (!camera.current) return; setIsRecording(true); camera.current.startRecording({ onRecordingFinished: (video: VideoFile) => { setIsRecording(false); setIsPaused(false); // video.path - temporary file path // video.duration - duration in seconds console.log("Recorded:", video.path, `${video.duration}s`); }, onRecordingError: (error) => { setIsRecording(false); setIsPaused(false); // Handle capture/recording-canceled gracefully if (error.code === "capture/recording-canceled") { console.log("Recording was canceled"); return; } console.error("Recording error:", error); }, videoCodec: "h265", flash: "off", }); }, []); const handleStopRecording = useCallback(async () => { await camera.current?.stopRecording(); }, []); const handleTogglePause = useCallback(async () => { if (isPaused) { await camera.current?.resumeRecording(); } else { await camera.current?.pauseRecording(); } setIsPaused((prev) => !prev); }, [isPaused]); if (device == null) return null; if (!hasMicPermission) { return ( <Pressable onPress={requestMicPermission}> <Text>Grant Microphone Permission</Text> </Pressable> ); } return ( <View style={styles.container}> <Camera ref={camera} style={StyleSheet.absoluteFill} device={device} isActive={true} video={true} audio={true} /> {!isRecording ? ( <Pressable onPress={handleStartRecording}><Text>Record</Text></Pressable> ) : ( <View> <Pressable onPress={handleTogglePause}> <Text>{isPaused ? "Resume" : "Pause"}</Text> </Pressable> <Pressable onPress={handleStopRecording}><Text>Stop</Text></Pressable> </View> )} </View> ); } const styles = StyleSheet.create({ container: { flex: 1 }, }); ``` **Why good:** microphone permission handled separately, recording state tracked, pause/resume supported, cancelRecording error code handled gracefully, h265 codec for better compression --- ## Pattern 4: HDR Format Selection and Location Metadata ```typescript import { Camera, useCameraDevice, useCameraFormat, useLocationPermission, } from "react-native-vision-camera"; const TARGET_FPS = 30; export function HDRCameraScreen() { const device = useCameraDevice("back"); const { hasPermission: hasLocationPermission, requestPermission: requestLocationPermission } = useLocationPermission(); // Format selection with HDR priority const format = useCameraFormat(device, [ { videoHdr: true }, { photoHdr: true }, { fps: TARGET_FPS }, ]); if (device == null) return null; return ( <Camera device={device} isActive={true} photo={true} video={true} format={format} videoHdr={format?.supportsVideoHdr} photoHdr={format?.supportsPhotoHdr} enableLocation={hasLocationPermission} fps={TARGET_FPS} /> ); } ``` **Why good:** HDR enabled only when format supports it (prevents crashes), location enabled only with permission, format priorities ordered by importance, named constant for FPS **Platform manifest requirements for location:** - **iOS**: Add `NSLocationWhenInUseUsageDescription` to Info.plist - **Android**: Add `ACCESS_FINE_LOCATION` to AndroidManifest.xml - **Build flag**: Set `VCEnableLocation = false` in Podfile/Expo config if location is unused (prevents App Store rejection) --- ## Pattern 5: Multi-Camera Device Selection ```typescript import { useCameraDevice, useCameraDevices } from "react-native-vision-camera"; import { useMemo } from "react"; // Simple: best back camera const backDevice = useCameraDevice("back"); // Multi-lens: prefer device with ultra-wide + wide + telephoto const multiLensDevice = useCameraDevice("back", { physicalDevices: [ "ultra-wide-angle-camera", "wide-angle-camera", "telephoto-camera", ], }); // Custom selection with all devices function useExternalCamera() { const devices = useCameraDevices(); return useMemo( () => devices.find((d) => d.position === "external") ?? null, [devices], ); } ``` **Physical device types and their zoom levels:** | Physical Device | Typical Zoom | | ------------------------- | ------------ | | `ultra-wide-angle-camera` | 0.5x | | `wide-angle-camera` | 1x (default) | | `telephoto-camera` | 3x | **Key:** `device.neutralZoom` gives the recommended starting zoom level for the selected device. On ultra-wide cameras this is NOT 1.0. -
scanning-and-processing.md 9 KB
# VisionCamera - Scanning and Frame Processing > Code scanning and real-time frame processor patterns. See [core.md](core.md) for camera setup and capture. See [SKILL.md](../SKILL.md) for red flags. --- ## Pattern 1: QR/Barcode Scanning with Debounce The code scanner fires many times per second. Guard against processing the same code repeatedly. ```typescript import { useCallback, useRef } from "react"; import { Alert, StyleSheet } from "react-native"; import { Camera, useCameraDevice, useCodeScanner } from "react-native-vision-camera"; const SCAN_COOLDOWN_MS = 2000; export function ScannerScreen() { const device = useCameraDevice("back"); const lastScannedRef = useRef<string | null>(null); const lastScannedTimeRef = useRef(0); const codeScanner = useCodeScanner({ codeTypes: ["qr", "ean-13", "code-128"], onCodeScanned: (codes) => { const code = codes[0]; if (!code?.value) return; const now = Date.now(); // Debounce: skip if same code scanned within cooldown if ( code.value === lastScannedRef.current && now - lastScannedTimeRef.current < SCAN_COOLDOWN_MS ) { return; } lastScannedRef.current = code.value; lastScannedTimeRef.current = now; Alert.alert("Scanned", code.value); }, }); if (device == null) return null; return ( <Camera style={StyleSheet.absoluteFill} device={device} isActive={true} codeScanner={codeScanner} /> ); } ``` **Why good:** ref-based debounce avoids re-renders, cooldown prevents duplicate processing, only needed code types listed (not all), named constant for cooldown ```typescript // Bad: no debounce, setState on every scan const codeScanner = useCodeScanner({ codeTypes: ["qr"], onCodeScanned: (codes) => { setScannedCode(codes[0]?.value); // Fires 30+ times per second }, }); ``` **Why bad:** setState called on every frame with a visible code, causes excessive re-renders and may trigger navigation/alerts repeatedly ### Supported Code Types | Code Type | Platform | Notes | | ------------- | ------------- | ------------------------------------------- | | `qr` | iOS + Android | QR codes | | `ean-13` | iOS + Android | European article numbers | | `ean-8` | iOS + Android | Short EAN | | `code-128` | iOS + Android | High-density barcode | | `code-39` | iOS + Android | Alphanumeric barcode | | `code-93` | iOS + Android | | | `upc-e` | iOS + Android | US product codes | | `pdf-417` | iOS + Android | 2D barcode (IDs, tickets) | | `aztec` | iOS + Android | 2D barcode | | `data-matrix` | iOS + Android | 2D matrix code | | `itf` | iOS + Android | Interleaved 2 of 5 (min 6 chars on Android) | | `codabar` | iOS + Android | | **Platform notes:** - UPC-A codes report as EAN-13 on iOS (EAN-13 is a superset of UPC-A) - ITF-14 (14-character restriction) is iOS-only - Android requires MLKit: set `VisionCamera_enableCodeScanner=true` in `gradle.properties` --- ## Pattern 2: Basic Frame Processor Frame processors run as worklets on a parallel JS thread. The `'worklet'` directive is mandatory. ```typescript import { useFrameProcessor, Camera, useCameraDevice } from "react-native-vision-camera"; import { StyleSheet } from "react-native"; export function FrameProcessorScreen() { const device = useCameraDevice("back"); const frameProcessor = useFrameProcessor((frame) => { "worklet"; // Access frame properties console.log(`Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})`); // Call native frame processor plugins // const results = detectFaces(frame); }, []); if (device == null) return null; return ( <Camera style={StyleSheet.absoluteFill} device={device} isActive={true} frameProcessor={frameProcessor} /> ); } ``` **Why good:** worklet directive present as first statement, dependency array provided, frame properties accessed synchronously ```typescript // Bad: missing worklet directive const frameProcessor = useFrameProcessor((frame) => { console.log(frame.width); // Will crash or run on wrong thread }, []); ``` **Why bad:** without `'worklet'` directive, function runs on the wrong thread, causing crashes or silent failures --- ## Pattern 3: Async Frame Processing with runAsync For heavy processing that exceeds the frame interval (~33ms at 30 FPS), use `runAsync` to avoid blocking the camera pipeline. ```typescript import { useFrameProcessor } from "react-native-vision-camera"; import { runAsync } from "react-native-vision-camera"; const frameProcessor = useFrameProcessor((frame) => { "worklet"; // Heavy work runs on a separate thread, non-blocking runAsync(frame, () => { "worklet"; // This won't block the camera pipeline // Only one runAsync runs at a time (not parallel) const results = heavyMLInference(frame); // Process results... }); }, []); ``` **Why good:** heavy work offloaded to separate thread, camera pipeline not blocked, frames continue flowing **Key:** `runAsync` runs one call at a time. If the previous async call is still running, new calls are skipped. This naturally throttles heavy work. --- ## Pattern 4: Throttled Processing with runAtTargetFps When you don't need to process every frame (e.g., ML detection at 5 FPS is sufficient): ```typescript import { useFrameProcessor } from "react-native-vision-camera"; import { runAtTargetFps } from "react-native-vision-camera"; const DETECTION_FPS = 5; const frameProcessor = useFrameProcessor((frame) => { "worklet"; // Process at 5 FPS instead of 30 runAtTargetFps(DETECTION_FPS, () => { "worklet"; const detections = detectObjects(frame); // Handle detections... }); }, []); ``` **Why good:** ML inference doesn't need 30 FPS, reduces battery and CPU usage significantly, named constant for target FPS --- ## Pattern 5: Sharing Data Between Frame Processor and React Use `useSharedValue` from Reanimated for efficient cross-thread communication. Use `createRunOnJS` to call React functions from worklets. ```typescript import { useCallback } from "react"; import { useFrameProcessor } from "react-native-vision-camera"; import { useSharedValue } from "react-native-reanimated"; import { createRunOnJS } from "react-native-vision-camera"; const DETECTION_FPS = 10; export function DetectionScreen() { // Shared value for cross-thread data (no re-renders) const detectedObjects = useSharedValue<DetectedObject[]>([]); // React function to call from worklet const handleDetection = useCallback((objects: DetectedObject[]) => { // This runs on the React JS thread if (objects.length > 0) { console.log("Detected:", objects.length, "objects"); } }, []); const frameProcessor = useFrameProcessor( (frame) => { "worklet"; runAtTargetFps(DETECTION_FPS, () => { "worklet"; const objects = detectObjects(frame); // Update shared value (accessible from Reanimated/Skia) detectedObjects.value = objects; // Call React function from worklet const onDetection = createRunOnJS(handleDetection); onDetection(objects); }); }, [handleDetection], ); // detectedObjects.value can be used in Reanimated animated styles // to overlay bounding boxes without re-rendering React components } ``` **Why good:** useSharedValue avoids useState re-renders, createRunOnJS bridges worklet-to-React safely, throttled at target FPS ```typescript // Bad: using useState from frame processor const [objects, setObjects] = useState<DetectedObject[]>([]); const frameProcessor = useFrameProcessor((frame) => { "worklet"; const result = detectObjects(frame); setObjects(result); // BAD: causes thread context switching + re-renders every frame }, []); ``` **Why bad:** useState causes React re-renders on every processed frame, thread context switching between worklet and React thread is expensive, UI may become unresponsive --- ## Pattern 6: Reading React State in Frame Processors React state values are automatically copied (read-only) into frame processor worklets via closure. ```typescript const [isDetectionEnabled, setIsDetectionEnabled] = useState(true); const frameProcessor = useFrameProcessor( (frame) => { "worklet"; // isDetectionEnabled is readonly-copied into the worklet if (!isDetectionEnabled) return; const results = detectObjects(frame); // Process results... }, [isDetectionEnabled], // Re-create processor when state changes ); ``` **Why good:** React state readable in worklets via closure, dependency array ensures processor updates when state changes **Key:** State values are read-only copies. You cannot set React state from within a worklet directly -- use `createRunOnJS` for that.
-
-
reference.md 10.3 KB
# VisionCamera Reference > Decision frameworks, device/format selection, and performance checklist. See [SKILL.md](SKILL.md) for red flags and anti-patterns. --- ## Device Selection Framework ``` Which camera position? | +-> Back camera (default) → useCameraDevice("back") +-> Front camera (selfie) → useCameraDevice("front") +-> Multi-lens back camera → useCameraDevice("back", { | physicalDevices: ["ultra-wide-angle-camera", "wide-angle-camera", "telephoto-camera"] | }) +-> External USB camera → useCameraDevices() + filter for position === "external" ``` **Physical device zoom levels:** | Device | Zoom | Use Case | | ------------------------- | ----- | --------------------------- | | `ultra-wide-angle-camera` | ~0.5x | Landscapes, group photos | | `wide-angle-camera` | 1x | General photography | | `telephoto-camera` | ~3x | Portraits, distant subjects | --- ## Format Selection Framework Filters are **ordered by descending priority** -- first filter has highest priority. ``` What matters most? | +-> Video quality → [{ videoResolution: { width: 3840, height: 2160 } }, { fps: 30 }] +-> Photo quality → [{ photoResolution: "max" }] +-> Performance → [{ videoResolution: { width: 1280, height: 720 } }, { fps: 30 }] +-> HDR → [{ videoHdr: true }, { videoResolution: { width: 1920, height: 1080 } }] +-> High FPS → [{ fps: 60 }, { videoResolution: { width: 1920, height: 1080 } }] ``` **Format properties:** | Property | Type | Description | | -------------------------- | -------- | ------------------------ | | `photoWidth`/`photoHeight` | number | Photo capture resolution | | `videoWidth`/`videoHeight` | number | Video/preview resolution | | `minFps`/`maxFps` | number | FPS range | | `supportsVideoHdr` | boolean | 10-bit HDR video | | `supportsPhotoHdr` | boolean | Multi-exposure HDR photo | | `supportsDepthCapture` | boolean | Depth data available | | `videoStabilizationModes` | string[] | Stabilization options | --- ## Zoom Implementation Zoom is **logarithmic** -- linear gesture input must be mapped to logarithmic zoom values. ```typescript // Animated zoom with Reanimated + Gesture Handler import Animated, { useSharedValue, useAnimatedProps, interpolate, Extrapolation, } from "react-native-reanimated"; import { Gesture, GestureDetector } from "react-native-gesture-handler"; const ReanimatedCamera = Animated.createAnimatedComponent(Camera); Animated.addWhitelistedNativeProps({ zoom: true }); const zoom = useSharedValue(device.neutralZoom); const MAX_ZOOM = 16; // Clamp to reasonable max const pinchGesture = Gesture.Pinch().onUpdate((event) => { zoom.value = interpolate( event.scale, [1, 10], [device.minZoom, Math.min(device.maxZoom, MAX_ZOOM)], Extrapolation.CLAMP, ); }); const animatedProps = useAnimatedProps(() => ({ zoom: zoom.value, })); <GestureDetector gesture={pinchGesture}> <ReanimatedCamera animatedProps={animatedProps} {...props} /> </GestureDetector> ``` **Simpler alternative:** Use `enableZoomGesture={true}` for built-in pinch-to-zoom without custom gesture code. --- ## Camera Props Quick Reference | Prop | Type | Default | Notes | | ------------------------- | ---------------------------------- | ----------- | -------------------------------------- | | `device` | CameraDevice | required | From `useCameraDevice()` | | `isActive` | boolean | required | Toggle, don't unmount | | `photo` | boolean | false | Enable photo capture pipeline | | `video` | boolean | false | Enable video recording pipeline | | `audio` | boolean | false | Enable audio (requires mic permission) | | `format` | CameraDeviceFormat | auto | From `useCameraFormat()` | | `fps` | number \| [min, max] | auto | Fixed or variable FPS | | `photoQualityBalance` | "speed" \| "balanced" \| "quality" | "balanced" | Photo capture speed vs quality | | `zoom` | number | neutralZoom | Logarithmic scale | | `exposure` | number | neutral | Offset from auto-exposure | | `videoHdr` | boolean | false | Requires format support | | `photoHdr` | boolean | false | Requires format support | | `enableLocation` | boolean | false | GPS metadata in captures | | `enableZoomGesture` | boolean | false | Built-in pinch-to-zoom | | `enableBufferCompression` | boolean | false | Lossy compression for less memory | | `videoStabilizationMode` | string | "off" | Check format support | | `codeScanner` | CodeScanner | - | From `useCodeScanner()` | | `frameProcessor` | FrameProcessor | - | From `useFrameProcessor()` | --- ## Camera Ref Methods | Method | Returns | Notes | | ------------------------- | -------------------- | -------------------------------------------- | | `takePhoto(options?)` | `Promise<PhotoFile>` | Full quality, AE/AF/AWB | | `takeSnapshot(options?)` | `Promise<PhotoFile>` | ~16ms, preview quality, needs `video` on iOS | | `startRecording(options)` | void | Callback-based (onRecordingFinished) | | `stopRecording()` | `Promise<void>` | Triggers onRecordingFinished | | `pauseRecording()` | `Promise<void>` | | | `resumeRecording()` | `Promise<void>` | | | `cancelRecording()` | `Promise<void>` | Deletes file, fires onRecordingError | | `focus(point)` | `Promise<void>` | { x, y } relative to view | --- ## Hooks Quick Reference | Hook | Returns | Purpose | | --------------------------------------- | -------------------------------------- | ------------------------------ | | `useCameraDevice(position, options?)` | `CameraDevice \| undefined` | Best device for position | | `useCameraDevices()` | `CameraDevice[]` | All available devices | | `useCameraFormat(device, filters)` | `CameraDeviceFormat \| undefined` | Best format matching filters | | `useCameraPermission()` | `{ hasPermission, requestPermission }` | Camera permission state | | `useMicrophonePermission()` | `{ hasPermission, requestPermission }` | Mic permission state | | `useLocationPermission()` | `{ hasPermission, requestPermission }` | Location permission state | | `useCodeScanner(options)` | `CodeScanner` | QR/barcode scanner config | | `useFrameProcessor(callback, deps)` | `FrameProcessor` | Worklet-based frame processing | | `useSkiaFrameProcessor(callback, deps)` | `FrameProcessor` | Skia canvas on frames | --- ## Camera Lifecycle Events Events fire in this order: 1. `onInitialized` -- session ready, all Camera methods available 2. `onStarted` / `onStopped` -- session streaming started/stopped 3. `onPreviewStarted` / `onPreviewStopped` -- preview frames flowing/stopped 4. `onError` -- runtime error occurred **Key:** Do NOT call `takePhoto()` or `startRecording()` before `onInitialized` fires. --- ## Performance Checklist ### Pipeline Optimization - [ ] Only enabling pipelines actually in use (`photo`, `video`, `codeScanner`, frame processor) - [ ] Using `enableBufferCompression={true}` when memory is a concern - [ ] Disabling `videoHdr` when 10-bit processing overhead is unnecessary - [ ] Disabling `videoStabilizationMode` if startup speed matters more ### Resolution and FPS - [ ] Format resolution matches actual need (not 4K when 1080p suffices) - [ ] FPS set to actual need (not 60 when 30 is fine) - [ ] Using variable FPS `fps={[minFps, maxFps]}` for adaptive low-light ### Frame Processor Performance - [ ] `'worklet'` directive present as first line of every frame processor - [ ] Using `runAsync` for processing > 33ms (at 30 FPS) - [ ] Using `runAtTargetFps` when per-frame processing is unnecessary - [ ] Using `useSharedValue` (not `useState`) for cross-thread data - [ ] Using native plugins over pure JS for heavy operations - [ ] Preferring YUV pixel format over RGB (less memory overhead) - [ ] Using `enableFpsGraph={true}` during development to profile performance ### Device Selection - [ ] Using simpler devices (fewer physical cameras) when multi-lens is unnecessary (faster init) - [ ] Starting zoom at `device.neutralZoom` (not hardcoded 1.0) ### Lifecycle - [ ] Camera mounted once, toggling `isActive` instead of unmounting - [ ] `isActive` combines screen focus AND app state - [ ] Not calling Camera methods before `onInitialized` --- ## Permission States | Status | Meaning | Action | | ---------------- | ------------------------ | ------------------------------------ | | `granted` | User approved | Render Camera | | `not-determined` | Never asked | Call `requestPermission()` | | `denied` | User rejected | Show rationale, link to Settings | | `restricted` | Device policy (parental) | Show explanation, no action possible | -
SKILL.md 16.9 KB
--- name: mobile-camera-vision-camera description: VisionCamera v4+ - photo/video capture, QR/barcode scanning, real-time frame processors, zoom/focus/exposure, HDR, location metadata, format selection --- # VisionCamera Patterns > **Quick Guide:** Use VisionCamera for high-performance camera features in React Native. Control the camera lifecycle with `isActive` (never unmount/remount). Use `useCameraDevice` to select back/front cameras, `useCameraPermission` for permissions. Capture photos with `takePhoto()`, record video with `startRecording()`/`stopRecording()`, scan codes with `useCodeScanner`, and process frames in real time with `useFrameProcessor` worklets. Frame processors run on a parallel JS thread via JSI -- keep them fast or use `runAsync`/`runAtTargetFps` to avoid blocking the pipeline. --- <critical_requirements> ## CRITICAL: Before Using This Skill > **All code must follow project conventions in CLAUDE.md** (kebab-case, named exports, import ordering, `import type`, named constants) **(You MUST set `isActive` based on screen focus AND app state -- camera must pause when backgrounded or navigated away)** **(You MUST request permissions before rendering the Camera -- `useCameraPermission` returns `hasPermission` and `requestPermission`)** **(You MUST include the `'worklet'` directive as the first line of every frame processor function body)** **(You MUST enable only the pipelines you need (`photo`, `video`, `codeScanner`, `frameProcessor`) -- unused pipelines waste resources)** **(You MUST use `useSharedValue` (not `useState`) for data shared between frame processors and the React thread)** </critical_requirements> --- **Auto-detection:** VisionCamera, react-native-vision-camera, useCameraDevice, useCameraDevices, useCameraPermission, useMicrophonePermission, useCodeScanner, useFrameProcessor, useSkiaFrameProcessor, useCameraFormat, takePhoto, takeSnapshot, startRecording, stopRecording, Camera component, frame processor, worklet, codeScanner, photoQualityBalance, enableLocation, videoHdr, photoHdr **When to use:** - Capturing photos or recording video in a React Native app - Scanning QR codes or barcodes (EAN-13, Code-128, etc.) - Real-time frame processing for ML, object detection, or image analysis - Implementing zoom, focus, exposure, or HDR controls - Embedding GPS location metadata in captured media - Selecting specific camera devices (ultra-wide, telephoto, front/back) **When NOT to use:** - Picking images from the device gallery (use an image picker) - Simple static image display (use standard Image component) - Web-only camera access (use browser MediaDevices API) **Key patterns covered:** - Camera lifecycle management with `isActive` and screen/app state - Permission handling with hooks (`useCameraPermission`, `useMicrophonePermission`) - Photo capture (`takePhoto`, `takeSnapshot`) and video recording - QR/barcode scanning with `useCodeScanner` - Frame processors with worklets, `runAsync`, and `runAtTargetFps` - Device selection, format selection, zoom, focus, exposure, HDR - Location metadata embedding - Performance optimization (pipeline selection, buffer compression, pixel format) **Detailed Resources:** - [examples/core.md](examples/core.md) - Camera setup, permissions, lifecycle, photo capture, video recording - [examples/scanning-and-processing.md](examples/scanning-and-processing.md) - Code scanning, frame processors, worklet patterns - [reference.md](reference.md) - Decision frameworks, device/format selection, performance checklist --- <philosophy> ## Philosophy VisionCamera provides direct, high-performance camera access in React Native via JSI (JavaScript Interface). It bypasses the legacy bridge entirely, giving synchronous control over native camera hardware from JavaScript. **Core principles:** 1. **Lifecycle-driven** -- the `isActive` prop controls the camera session. Toggle it instead of mounting/unmounting. Resuming is much faster than re-mounting. 2. **Pipeline-based** -- enable only what you need (`photo`, `video`, `codeScanner`, frame processor). Each pipeline allocates resources. 3. **Worklet-powered** -- frame processors run on a parallel JS thread via `react-native-worklets-core`. They execute synchronously in the video pipeline, so they must be fast. 4. **Device/format-aware** -- different physical cameras and formats have different capabilities. Always check device and format properties before enabling features like HDR or high FPS. **When to use VisionCamera:** - You need camera preview with capture, scanning, or real-time processing - You need fine-grained control over device, format, zoom, focus, exposure - You need frame-level access for ML inference or custom image processing **When NOT to use:** - Gallery/file picking (different concern entirely) - Screenshot or screen recording (not camera-related) </philosophy> --- <patterns> ## Core Patterns ### Pattern 1: Camera Lifecycle and Permissions The camera must be activated only when the screen is focused AND the app is in the foreground. Always check permissions before rendering. ```typescript import { useCameraDevice, useCameraPermission, Camera } from "react-native-vision-camera"; import { StyleSheet } from "react-native"; export function CameraScreen() { const device = useCameraDevice("back"); const { hasPermission, requestPermission } = useCameraPermission(); // Request permission on mount if not granted // Render permission UI or Camera based on hasPermission if (!hasPermission) return <PermissionRequest onRequest={requestPermission} />; if (device == null) return <NoCameraDeviceError />; return <Camera style={StyleSheet.absoluteFill} device={device} isActive={isActive} />; } ``` **Why good:** permission checked before render, device null-checked, isActive controls lifecycle without unmounting The `isActive` prop should combine screen focus and app state: ```typescript const isFocused = useIsFocused(); // from navigation const appState = useAppState(); // from community hooks const isActive = isFocused && appState === "active"; ``` See [examples/core.md](examples/core.md) for the complete lifecycle pattern with permissions, device selection, and app state handling. --- ### Pattern 2: Photo Capture Use a Camera ref to call `takePhoto()`. Enable the `photo` pipeline on the Camera component. Use `takeSnapshot()` for fast preview-quality captures. ```typescript const camera = useRef<Camera>(null); const handleCapture = async () => { const photo = await camera.current?.takePhoto({ flash: "auto", enableShutterSound: true, }); // photo.path contains the temporary file path }; <Camera ref={camera} device={device} isActive={isActive} photo={true} /> ``` **Why good:** photo pipeline explicitly enabled, ref used for imperative capture, options typed - `takePhoto()` -- full-quality capture with AE/AF/AWB, supports flash - `takeSnapshot()` -- ~16ms capture from preview buffer, requires `video` enabled on iOS - `photoQualityBalance` prop: `"speed"` | `"balanced"` | `"quality"` See [examples/core.md](examples/core.md) for full photo capture with error handling and format selection. --- ### Pattern 3: Video Recording Use `startRecording()` with callbacks for completion and errors. `stopRecording()` triggers `onRecordingFinished`. ```typescript camera.current?.startRecording({ onRecordingFinished: (video) => { // video.path contains the temporary file path // video.duration contains the duration in seconds }, onRecordingError: (error) => console.error(error), }); // Later: await camera.current?.stopRecording(); ``` **Why good:** callback-based API handles async completion, error callback prevents silent failures - Enable `video={true}` and `audio={true}` on the Camera component - `pauseRecording()` / `resumeRecording()` for pause support - `cancelRecording()` deletes the temp file and fires `onRecordingError` - `videoBitRate`: `"low"` | `"normal"` | `"high"` or custom Mbps number - `videoCodec`: `"h264"` (default, wider compatibility) or `"h265"` (better compression) See [examples/core.md](examples/core.md) for complete video recording with pause/resume. --- ### Pattern 4: QR/Barcode Scanning Use the `useCodeScanner` hook. Runs on the native thread for instant scanning without UI freezing. ```typescript const codeScanner = useCodeScanner({ codeTypes: ["qr", "ean-13"], onCodeScanned: (codes) => { // codes[0].value contains the decoded string // Fires many times per second -- debounce if updating state }, }); <Camera device={device} isActive={isActive} codeScanner={codeScanner} /> ``` **Why good:** native-thread scanning, specific code types listed (not all), callback provides decoded values **Android setup:** Requires MLKit. Enable via `VisionCamera_enableCodeScanner=true` in `gradle.properties` (or Expo plugin config). **Gotcha:** `onCodeScanned` fires many times per second. Debounce or guard with a ref to avoid processing the same code repeatedly. See [examples/scanning-and-processing.md](examples/scanning-and-processing.md) for debounced scanning and supported code types. --- ### Pattern 5: Frame Processors Frame processors run JavaScript worklets on a parallel camera thread for real-time frame analysis. Requires `react-native-worklets-core`. ```typescript const frameProcessor = useFrameProcessor((frame) => { "worklet"; // frame.width, frame.height, frame.pixelFormat // frame.toArrayBuffer() for raw pixel data const results = detectObjects(frame); // native plugin call }, []); <Camera device={device} isActive={isActive} frameProcessor={frameProcessor} /> ``` **Why good:** worklet directive present, runs on parallel thread, native plugin for heavy work - Synchronous by default -- must complete before next frame (~33ms at 30 FPS) - `runAsync(() => { ... })` -- offload heavy processing without blocking - `runAtTargetFps(5, () => { ... })` -- process at lower FPS to save resources - `useSharedValue` -- share data between frame processor and React thread - `createRunOnJS(fn)` -- call React functions from within a worklet See [examples/scanning-and-processing.md](examples/scanning-and-processing.md) for async processing, shared values, and performance patterns. --- ### Pattern 6: Device and Format Selection Choose camera devices by position and physical lenses. Select formats for resolution, FPS, and HDR support. ```typescript // Basic device selection const device = useCameraDevice("back"); // Multi-camera with specific lenses const device = useCameraDevice("back", { physicalDevices: [ "ultra-wide-angle-camera", "wide-angle-camera", "telephoto-camera", ], }); // Format selection (filters ordered by descending priority) const format = useCameraFormat(device, [ { videoAspectRatio: 16 / 9 }, { videoResolution: { width: 1920, height: 1080 } }, { fps: 30 }, ]); ``` **Why good:** multi-camera support with fallback, format priorities explicitly ordered, resolution matched to actual need See [reference.md](reference.md) for the device and format decision framework. --- ### Pattern 7: Zoom, Focus, and Exposure Control camera optics via props and ref methods. ```typescript // Zoom: use device.minZoom, device.maxZoom, device.neutralZoom <Camera zoom={device.neutralZoom} /> // Focus: tap-to-focus via ref await camera.current?.focus({ x: tapX, y: tapY }); // Exposure: offset from auto-exposure (-2 to +2 typical range) <Camera exposure={exposureOffset} /> ``` - Zoom operates on a **logarithmic scale** -- use `interpolate()` for linear gesture mapping - Built-in pinch-to-zoom: `enableZoomGesture={true}` (no custom gesture needed) - Focus adjusts both AF and AE at the tap point - Check `device.supportsFocus` before calling `focus()` - Exposure range: `device.minExposure` to `device.maxExposure` See [reference.md](reference.md) for animated zoom with Reanimated. --- ### Pattern 8: HDR and Location Metadata Enable HDR for enhanced dynamic range. Enable location for GPS EXIF/MP4 tags. ```typescript const format = useCameraFormat(device, [ { videoHdr: true }, { photoHdr: true }, ]); <Camera format={format} videoHdr={format?.supportsVideoHdr} photoHdr={format?.supportsPhotoHdr} enableLocation={true} /> ``` - Video HDR uses 10-bit pixel format (adds processing overhead) - Photo HDR combines multiple exposures into one image - Location requires `useLocationPermission` and platform-specific manifest entries - Disable location APIs entirely with build flag if not needed (avoids App Store rejection) See [examples/core.md](examples/core.md) for HDR format selection and location permission flow. </patterns> --- <decision_framework> ## Decision Framework ### Capture Method ``` What do you need to capture? | +-> Still image? | +-> High quality (AE/AF/AWB) → takePhoto() | +-> Fast preview capture (~16ms) → takeSnapshot() (requires video pipeline) | +-> Video? | +-> Continuous recording → startRecording() / stopRecording() | +-> Need pause/resume → pauseRecording() / resumeRecording() | +-> User cancelled → cancelRecording() | +-> QR/barcode scanning? | +-> useCodeScanner (native thread, no frame processor needed) | +-> Real-time frame analysis (ML, detection)? +-> useFrameProcessor with native plugins +-> Need Skia drawing on frames? → useSkiaFrameProcessor ``` ### Frame Processor Scheduling ``` How fast must your processor run? | +-> Every frame (30/60 FPS)? | +-> Processing < 33ms? → Default synchronous processor | +-> Processing > 33ms? → runAsync (offload to separate thread) | +-> Lower rate is fine (5-10 FPS)? +-> runAtTargetFps(targetFps, () => { ... }) ``` ### Device Selection ``` Which camera? | +-> Simple back/front → useCameraDevice("back") or useCameraDevice("front") +-> Multi-lens (0.5x + 1x + 3x) → useCameraDevice("back", { physicalDevices: [...] }) +-> External USB camera → Filter useCameraDevices() for position === "external" +-> Custom logic → useCameraDevices() + useMemo with your own filter ``` </decision_framework> --- <red_flags> ## RED FLAGS **High Priority Issues:** - Mounting/unmounting Camera instead of toggling `isActive` -- wastes resources, slow resume - Missing `'worklet'` directive in frame processor function -- silently runs on wrong thread, crashes - Using `useState` to share data from frame processors -- causes thread context switching, use `useSharedValue` - Enabling all pipelines (`photo`, `video`, `codeScanner`, frame processor) when only one is needed -- wastes memory and battery - Not requesting permissions before rendering Camera -- causes crash or blank preview - Calling `camera.current.takePhoto()` before `onInitialized` fires -- method not ready, throws error **Medium Priority Issues:** - Capturing 4K when 1080p is sufficient -- wastes memory, slower processing - Not debouncing `onCodeScanned` -- fires many times per second, causes excessive state updates - Using `useSkiaFrameProcessor` when `useFrameProcessor` suffices -- Skia adds overhead - Enabling `videoHdr` without checking `format.supportsVideoHdr` -- crashes on unsupported formats - Not checking `device.supportsFocus` before calling `focus()` -- fails on devices without AF **Gotchas & Edge Cases:** - Zoom is **logarithmic** -- 1x to 2x is a much bigger visual change than 127x to 128x. Use `interpolate()` for linear gesture mapping. - `takeSnapshot()` requires `video={true}` on iOS -- it captures from the video preview buffer - Frame processors at 4K process ~12MB per frame -- use lower resolution if possible - `device.neutralZoom` may not be 1.0 on ultra-wide cameras -- always use it as the starting zoom - Android code scanner requires MLKit (`VisionCamera_enableCodeScanner=true`) -- without it, scanning silently does nothing - UPC-A codes report as EAN-13 on iOS (EAN-13 is a superset) - `cancelRecording()` fires `onRecordingError` with `capture/recording-canceled` -- handle this error type gracefully - `enableLocation` requires platform manifest entries (iOS: `NSLocationWhenInUseUsageDescription`, Android: `ACCESS_FINE_LOCATION`) - Disable location APIs entirely via build flag if unused -- prevents App Store rejection for unnecessary privacy APIs - `exposure` prop is an offset from auto-exposure, not an absolute ISO value - Video HDR uses 10-bit pixel format which adds processing overhead -- disable when not needed </red_flags> --- <critical_reminders> ## CRITICAL REMINDERS > **All code must follow project conventions in CLAUDE.md** **(You MUST set `isActive` based on screen focus AND app state -- camera must pause when backgrounded or navigated away)** **(You MUST request permissions before rendering the Camera -- `useCameraPermission` returns `hasPermission` and `requestPermission`)** **(You MUST include the `'worklet'` directive as the first line of every frame processor function body)** **(You MUST enable only the pipelines you need (`photo`, `video`, `codeScanner`, `frameProcessor`) -- unused pipelines waste resources)** **(You MUST use `useSharedValue` (not `useState`) for data shared between frame processors and the React thread)** **Failure to follow these rules will cause crashes, blank previews, wasted battery, and dropped frames.** </critical_reminders>
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.