Camera 2.0 in Android 4.2
-
Upload
balwinder-kaur -
Category
Technology
-
view
6.312 -
download
0
description
Transcript of Camera 2.0 in Android 4.2
| © 2013 Aptina Imaging Corporation1
© 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners.
Camera 2.0 The New Camera Hardware Interface in Android 4.2
Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging
Ashutosh Gupta, Principal Software Engineer, Aptina Imaging
Android Builder’s Summit, San Francisco, CA
2.18.2013
| © 2013 Aptina Imaging Corporation2
Agenda• Camera Use Cases
‣ Prominent Use Cases
‣ Limitations of existing APIs
‣ Overview of android.hardware.Camera
• Android Framework : Camera Service
‣ Native Camera service
‣ New - Under the hood – Camera 2.0
• It’s all about the Camera hardware !
‣ Camera Hardware Abstraction Layer
‣ Camera Device Driver
• Challenges in Camera HAL development
• Emerging Trends
• Q&A
| © 2013 Aptina Imaging Corporation3
Section I
Android Camera APIs
| © 2013 Aptina Imaging Corporation4
Prominent Camera Use Cases
• Main Use Cases
‣ Live Preview of Camera Stream
• Live Preview + copy of the Frame returned to the application
‣ Capture a frame
‣ Video Recording of a Camera Stream
• Secondary Use Cases
‣ Configuring the Camera
‣ Snapshot during video recording
‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved
• Information Related Use cases• Receiving more than an image back . e.g. face detection data
• Meta Data of an Image
| © 2013 Aptina Imaging Corporation5
Limitations in Camera API
• No support for Burst Mode Photography
• No support for Panoramic Shots
• Very limited support for frame metadata
• No per-frame control of the camera or the image processing pipeline
• No access to control subsystems within the camera e.g. Focus, Flash, Image Sensor
| © 2013 Aptina Imaging Corporation6
Overview of android.hardware.Camera
6 Classes
• Camera
• Camera.CameraInfo
• Camera.Parameters
• Camera.Size
• Camera.Face
• Camera.Area
8 Callback Interfaces
• Camera.AutoFocusCallback
• Camera.ErrorCallback
• Camera.FaceDetectionListener
• Camera.OnZoomChangeListener
• Camera.PictureCallback
• Camera.PreviewCallback
• Camera.ShutterCallback
• Camera.AutoFocusMoveCallback
| © 2013 Aptina Imaging Corporation7
Camera class
Contains all the methods for the Camera Lifecycle
• Open & Release
• Access to the Camera Controls
• Preview
‣ Direct Live Preview to the display or a texture
‣ Get Preview Frame in a Callback
• Capture
‣ Callbacks: Shutter, JPEG, RAW, “Postview”
• Lock & Unlock
• Actions: startAutoFocus, startSmoothZoom & startFaceDetection
| © 2013 Aptina Imaging Corporation8
Camera.Parameters
Class for Camera Controls
① Mandatory Feature Set
‣ getSupportedPreviewSizes + set/get
② Optional Feature Set
‣ isVideoStabilizationSupported + set/get
③ Custom Feature Set
‣ Camera.Parameters class provides a “dumb” pipe to the hardware for custom controls
‣ set (“your_param_string”, value); get(“your_param_string”);
Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes, Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding
| © 2013 Aptina Imaging Corporation9
… the rest of the Camera Classes
• Camera.CameraInfo
‣ For each camera, front or back facing, orientation of the camera image
• Camera.Size
‣ width and height of the image
• Camera.Face
‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of the face
• Camera.Area
‣ Rectangular bounds with a weight
‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto Exposure
| © 2013 Aptina Imaging Corporation10
Camera 2 Internals
Released as part of Android 4.2
Camera 1 => android.hardware.Camera
Camera 2 => android.hardware.ProCamera
New Camera HAL implementation based on Camera 2.0 – Samsung exynos5 based
No application that actually uses Camera 2.0Disclaimer: All reference to Camera2 in this slide deck is based on reverse engineering AOSP code !
| © 2013 Aptina Imaging Corporation11
Section II
Android Framework - Camera
| © 2013 Aptina Imaging Corporation12
High Level Architecture
| © 2013 Aptina Imaging Corporation13
Source: Android Anatomy and Physiology, Google IO 2008
Android High Level Architecture
| © 2013 Aptina Imaging Corporation14
Source: Android Anatomy and Physiology, Google IO 2008
Hardware Abstraction Layer
Camera
| © 2013 Aptina Imaging Corporation15
Camera Subsystem
Application
Application framework
Camera Service
Camera HALImplementation
Camera Device Driver
Hardware Independent
Hardware Dependent
Camera Hardware
HAL = Hardware Abstraction Layer
| © 2013 Aptina Imaging Corporation16
Process View
App AppApp
Binder IPCICamera
Media server
Camera Service
Back Facing Camera
Hardware Object
Front Facing Camera
Hardware Object
Kernel Driver Kernel Driver
System Call System Call
Binder IPCISurface
Surfaceflinger
| © 2013 Aptina Imaging Corporation17
Inside the Camera App
Media server
Camera Servicelibcameraservice.so
Application Code
android framework code Camera.java
JNI
android_hardware_Camera.cpp
IBinder Interfaces
| © 2013 Aptina Imaging Corporation18
JNI Layer
| © 2013 Aptina Imaging Corporation19
android_hardware_Camera
• Allocates Memory from the Java memory heap for JPEG images.
• If a Copy of the Preview Frame is requested by the app, then the copy from native to java buffers is done here.
• Creates a persistent context for callbacks from native code to Java (JNICameraContext)
• Holds references to the Java Camera, Face and Area objects.
| © 2013 Aptina Imaging Corporation20
Camera Service
- with Camera 2.0
| © 2013 Aptina Imaging Corporation21
Camera Service
Media server
Camera Servicelibcameraservice.so
Camera HALimplementation
Application Code
android framework code Camera.java
JNI
android_hardware_Camera.cpp
ICameraClient | ICameraService | ICamera
camera.h
| © 2013 Aptina Imaging Corporation22
Camera Service with Camera 2
Media server
Camera Service libcameraservice.so
Camera HALimplementation
Application Code
android framework code Camera.java
JNI
android_hardware_Camera.cpp
ICameraClient | ICameraService | ICamera
camera.h
Camera2 HALimplementation
camera2.h
Camera2Client
Camera2Device
| © 2013 Aptina Imaging Corporation23
Camera Service
• Runs in the media server process
• It is a shared library libcameraservice.so
• Queries System for Camera HAL Api version - accordingly creates a CameraClient or Camera2Client for the application
• For Camera 1, libcameraservice does:Permission check android.permission.CAMERA
Ensures only one Client connects to a Camera Hardware Object
Ensures each Process connects to a single Camera Hardware Object
Redirects callbacks back to the app layer
Accessed over IBinder Interface
Number of Cameras Available
CameraInfo Details
| © 2013 Aptina Imaging Corporation24
Camera2Client• Implements android.hardware.camera API on top of camera device HAL
version 2
• Has processors that run in their own threads
Streaming Processor
JPEG Processor
Callback Processor
ZSL Processor
Frame Processor
Capture Sequencer
Preview Stream
Recording Stream
Callback Stream
Capture Stream
ZSL Stream
Manages Preview & Recording Streams
Still Image capture output processing
Processes Callback
ZSL Queue Processing
Metadata processing for output frames
Manages Capture Sequences for ZSLregular and Burst modes
| © 2013 Aptina Imaging Corporation25
Camera2Device
• Manages Input and Output Streams between the Camera HAL and the Service
• Functions
‣ Retrieves Static Metadata Information of the Camera
‣ Sends requests for Capture and Streaming
‣ Manages input and output streams including re-processing streams and metadata buffers
‣ StreamAdaptor between ANativeWindow Interface and the Camera HAL stream ops
• ANativeWindows is Android Native Window interface
‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and AWB)
| © 2013 Aptina Imaging Corporation26
Metadata• Camera2 has a lot of emphasis on retrieving metadata from the camera
and making it available to the application
• Two kinds of Metadata
‣ Static & Frame-Based
‣ Static does not change and is available without opening the camera device (Camera Info Class)
‣ *_INFO is the static metadata
ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO, ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO, ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO, ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC, ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP, ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER, ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS, ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO, ANDROID_QUIRKS_INFO etc.
• Extensible - there is provision for Vendor Specific Tags
| © 2013 Aptina Imaging Corporation27
Android Open Source Project (AOSP) Structure
• Android Framework
‣ Java: frameworks/base/core/java/android/hardware
‣ JNI: frameworks/base/core/jni
• Camera Service
‣ frameworks/av/services/camera/libcameraservice/
• IBinder Interfaces
‣ frameworks/av/include/camera/ICamera.h etc.
• IBinder Implementation
‣ frameworks/av/camera/ICamera.cpp etc.
• Camera HAL Interface
‣ hardware/libhardware/include/hardware/camera2.h etc.
• Camera HAL Implementation
‣ hardware/<vendor>/camera (typically)
• Camera Metadata
‣ system/media/camera/include/system/camera_metadata_tags.h
| © 2013 Aptina Imaging Corporation28
Section III
Its all about the Camera Hardware !
| © 2013 Aptina Imaging Corporation29
Camera Hardware Abstraction Layer
Review of a Typical Implementation
| © 2013 Aptina Imaging Corporation30
Camera Stack – Camera HAL
Camera Hardware Abstraction Layer
(HAL)
Vendor Specific HAL Implementation
Camera Driver
Image Sensor Image Sensor Processer
SurfaceFlinger /Overlay Buffers User
Kernel
Hardware
Upper Camera Stack …
| © 2013 Aptina Imaging Corporation31
Android CameraHAL Library
• The Camera Hardware Abstraction Layer (HAL) is a library that is specific to the camera hardware platform
‣ Written by hardware vendors (Qualcomm, Samsung, TI, others)
• CameraHAL maps Android Camera Service calls to driver functions
‣ Ice Cream Sandwich (ICS) uses camera.h
‣ Jelly Bean (and above) use camera2.h
• CameraHAL low level interface communicates with the kernel level driver
‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax (OMX)
‣ Communicates with the driver through file I/O calls (open, close, input/output controls (IOCTL), etc)
| © 2013 Aptina Imaging Corporation32
Sample ICS CameraHALFunctional Diagram
CameraHAL
Memory Manager
Camera Driver/dev/videoX
Camera Service I/F
Display Surface Manager
EventNotification Manager
Camera Manager
Kernel
User
Source: TI OMAP4 git.omapzoom.org
| © 2013 Aptina Imaging Corporation33
Sample JB CameraHALFunctional Diagram
CameraHAL Operations Handler
Stream Manager
Camera Driver/dev/videoX
Camera Service I/F
Metadata Handler
Reprocess Stream Management
Kernel
User
Source: Samsung Exynos 5
| © 2013 Aptina Imaging Corporation34
JB CameraHAL Block Diagram Discussion
• Block Functions
‣ CameraHAL
• Initialization, thread creation, function dispatch
‣ Stream Manager
• Handle stream events, coordinate buffer and stream usage, manage state machine
‣ Metadata Handler
• Acquire per shot metadata, convert to Android format
‣ Reprocess Stream Management
• Setup and manage reprocess streams
| © 2013 Aptina Imaging Corporation35
JB CameraHAL Changes
• For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is moved to libcameraservice
• CameraHAL 2.0 targeted at HW specific functionality
• New CameraHAL functionality currently not passed up to applications
• Image metadata has more importance
• Reprocessing introduced – Process an already captured image stream
• Stream based rather than stream function based
‣ CameraHAL 1.0 has methods for specific stream types (e.g., start_preview(), take_picture())
‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream())
| © 2013 Aptina Imaging Corporation36
Camera Device Driver
| © 2013 Aptina Imaging Corporation37
Camera Stack – Camera Driver
Camera Hardware Abstraction Layer
(HAL)
Vendor Specific HAL Implementation
Camera Driver
Image Sensor Image Sensor Processer
SurfaceFlinger /Overlay Buffers User
Kernel
Hardware
Upper Camera Stack …
| © 2013 Aptina Imaging Corporation38
Android Kernel Camera Driver
• The kernel driver presents a standard interface for different types of camera hardware
‣ Camera hardware specific attributes are handled by the low level kernel driver
‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor - differences are handled at the driver level
• For Android, Video for Linux 2 (V4L2) is used in many implementations
‣ V4L2 has been in existence for many years
‣ OpenMax (OMX) is also used for a low level driver interface by some vendors.
| © 2013 Aptina Imaging Corporation39
V4L2 Kernel Driver Block Diagram
V4L2 Driver Interface- IOCTL support/dispatch- V4L2 driver infrastructure
Controlling Interface- Support for different device configurations- Control device flow
Buffer/Memory Management- Memory allocation (if needed)- Buffer management- Buffer queue/de-queue
Camera HW Management- One of these blocks for each
camera type- Device discovery- Device initialization- Power management- Set/get device specific parameter- Enable/disable image streaming
| © 2013 Aptina Imaging Corporation40
V4L2 Kernel Driver Resources• Memory
‣ Memory can be either driver-allocated or user-provided
‣ The image transfers from the camera to memory through hardware Direct Memory Access (DMA)
‣ Hardware memory management may be used to avoid contiguous memory requirement
• Interrupts
‣ Camera ports support for interrupts on events such as frame start, finish, focus events, etc.
• Camera Control: I2C/SPI
‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers
‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C
• Control Signals/GPIO
‣ All controlled by the low level driver
• Power
‣ Sensor power management is critical to embedded device operation
‣ Sensors support standby mode where settings are maintained while power usage is reduced
| © 2013 Aptina Imaging Corporation41
V4L2 Driver Buffer Management• One or more buffers are supported
• User buffers or kernel-allocated buffers are supported
• Buffers are treated the same for preview, capture, video (output resolution does not matter)
• Buffers are queued to a circular list
• Buffer filling starts when the V4L2 Stream_On command is executed
• Once filled, the CameraHAL de-queues a buffer, processes the buffer, then re-queues the buffer
• The Stream_Off command causes all buffer to be released
Buff_0
Buff_1
Buff_2
Buff_3
Buff_x
| © 2013 Aptina Imaging Corporation42
Typical V4L2 Preview Sequence (1)
• V4L2 preview start up sequence is given below
V4L2 Call Driver Events Hardware Events
VIDIOC_S_FMT – set format
Set image format and size Set both resolution and output pixel format
VIDIOC_G_PARM – get parameter
Get a camera driver or hardware parameter
Read camera parameter
VIDIOC_S_PARM – set parameter
Set a camera driver or hardware parameter
Write camera parameter
VIDIOC_CROPCAP – get cropping capabilities
Return camera cropping capabilities
None
VIDIOC_S_CROP – set cropping
Set cropping rectangle Set camera cropping rectangle
VIDIOC_REQBUFS – request camera buffer
Request buffer support from the driver (user vs. kernel)
None
Loop: VIDIOC_QUERYBUF – query buffer caps
For kernel allocated buffers, return buffer characteristics
None
V4L2_MMAP – map buffers to user space
For kernel allocated buffers, memory map to user space
None
| © 2013 Aptina Imaging Corporation43
Typical V4L2 Preview Sequence (2)
• V4L2 preview start up sequence (cont)
• V4L preview shut-down sequence
V4L2 Call Driver Events Hardware Events
Loop: VIDIOC_QBUF – queue buffers
Queue buffers in the circular queue
none
VIDIOC_STREAM_ON – start streaming
Start image capture state Enable image output
V4L2 Call Driver Events Hardware Events
VIDIOC_STREAM_OFF – stop streaming
Stop streaming, deallocate buffer
Disable image output
| © 2013 Aptina Imaging Corporation44
V4L2 Media Controller Architecture
• Designed to support dynamically re-connectable hardware blocks.
• Allows for greater programmer control
• Introduces the notion of entities, pads and links
| © 2013 Aptina Imaging Corporation45
V4L2 Media Controller Architecture(Cont.)
| © 2013 Aptina Imaging Corporation46
Section IV
Challenges in Camera HAL Development
| © 2013 Aptina Imaging Corporation47
Challenges in Camera HAL Development
• Memory Management
• Various Implementation of Camera Driver
• Color Format Conversion
• Buffer Synchronization between Camera & Display
• Support for Advanced Features
| © 2013 Aptina Imaging Corporation48
Are all Camera HALs equal ?
• Supported Features depend on hardware capabilities
• Supported Features depend on the implementation in the Camera HAL
• Performance
• Reliability
• Extensions to the standard Android Feature Set
No
| © 2013 Aptina Imaging Corporation49
A Peek into the Future
| © 2013 Aptina Imaging Corporation50
Emerging Trends
• Computer Vision Applications go mainstream
‣ APIs on Object Tracking, Gesture Recognition become more common place
• Computation Photography application
‣ Developers get fine grained control of flash and camera
• High Dynamic Range
‣ Ability to capture larger exposure range
• 3D Imaging
‣ Use of 2 cameras to generate a 3D image
| © 2013 Aptina Imaging Corporation51
What’s coming ?
| © 2013 Aptina Imaging Corporation52
Q&A
| © 2013 Aptina Imaging Corporation53
References
• http://developer.android.com/
• http://www.codeaurora.org
• http://omappedia.org
• http://source.android.com
• http://stackoverflow.com/questions/10775942/android-sdk-get-raw-preview-camera-image-without-displaying-it
• http://www.cjontechnology.com/blog/?p=14
• http://graphics.stanford.edu/projects/camera-2.0/
| © 2013 Aptina Imaging Corporation54
About
Aptina enables imaging everywhere.
A leading innovator of CMOS imaging technology, Aptina delivers excellent pixel performance, sensor functionality and camera system capability to a world going visual.
Aptina Camera Software Stack (Access) is Aptina’s implementation of the Android Camera HAL with some custom extensions.
Contact Information
Balwinder Kaur [email protected]
Ashutosh Gupta [email protected]