Dongsoo Kim ([email protected])Heungjun Kim([email protected])
Samsung Electronics
2
Introducing mobile camera devices- SoC camera subsystem
- Mobile camera module devices
Framework for digital camera- Video4Linux2 and camera device
- New APIs for high-end mobile camera devices
- Dual camera framework for 3G handset
Porting issues in SoC camera subsystem
3
AP : Application Processor
Camera interface : interface device which is syncing image data from external camera device and gives some commands to them
Camera module : external camera device packaged with lens, sensor and ISP on it
ISP : Image Signal Processor. Also supports lots of additional functions
V4L2 : Video for Linux 2
4
This document contains some animations.
Slideshow mode is recommended
5
Exposure control- Aperture (Iris)
- Shutter
- ISO Lens control- Focus
- Zoom
Strobe support- Various program
mode : slow sync, red-eye reduction and on….
6
Introducing mobile camera devices
7
History of camera phone
First Camera Phone in 2000 with 3.5MP
1.3MP in 20032MP in 20043MP in20045MP in20047MP in 200510MP in 200612MP in 2009
8
Old and low-end mobile camera modules
Feature Low end camera
Exposure
Aperture Not supported
Shutter Just change frame rate
ISO Low sensitivity & High noise level
Lens
Focus Pan focus
Zoom Digital zoom
Strobe Just LED light most of time
9
Brand new and high-end mobile camera modulesFeature High end camera
Exposure
Aperture Supported with lens packaging
Shutter Supported with lens packaging
ISO High sensitivity & Low noise level
Lens
Focus Movable focus point (Auto/manual)
Zoom Optical zoom & Digital zoom
Strobe XENON flash is also supported
10
What is SoC camera subsystem?
ARM SoC solution
ARMCore
Peripheral1(Display)
Peripheral2 (MMC)
Peripheral3 (USB)Camera interface
Peripherals
External
Camera module
External
Camera module
11
Camera interface
Buffer handlingBuffer
handling
Resolution
handling
Resolution
handling
Data format handling
(YUV/RAW/JPEG…)
Data format handling
(YUV/RAW/JPEG…)
Camera module
I2CI2C
Exposure
control
Exposure
control
Lenscontr
ol
Lenscontr
ol
Strobe
control
Strobe
control
White balanc
e control
White balanc
e control
Effect contr
ol
Effect contr
ol
Facedetec
t
Facedetec
t
Zoomcontr
ol
Zoomcontr
ol
.
.
.
.
.
.
CommandinterfaceCommandinterface
RAW DATA
Image data in expected formatRequest for data
12
Various ways of preview (Live view) in SoC camera subsystems
CameraModule
Memory
CameraInterfa
ce
ARMcore
Frame buffer
OKCanc
el
User Applicati
onCOPY!
DATA
DA
TA
13
Various ways of preview (Live view) in SoC camera subsystems
CameraModule
Memory
CameraInterfa
ce
ARMcore
Frame buffer
OKCanc
el
User Applicati
on
COPY!
DMA Through
DMA
DATA
DA
TA
Saveas file
14
How to capture still shots in SoC camera subsystem- Your camera module has a dedicated JPEG encoder
on it
Memory
CameraInterfa
ce
ARMcore
User Applicati
onCOPY!
DATAMEMORY
Camera module
RAWDATA
PROCESS
JPEGPROCES
S
DATA
DA
TA
Save as JPEG file
15
How to capture still shots in SoC camera subsystem- No JPEG encoder in your camera module
CameraModule
Memory
CameraInterfa
ce
ARMcore
User Applicati
onCOPY!
DATA
DA
TA
RAWDATA
Codec S/W
Save as JPEG file
16
How to take motion pictures in SoC camera subsystem
CameraModule
Memory
CameraInterfa
ce
ARMcore
Frame buffer
OKCanc
el
User Applicati
on
COPY!
DATA
DA
TA
COPY!
Codec S/W
Save as…
17
In Kernel
V4L2 (AKA Video For Linux 2)- The second version of the Video For Linux API
- Kernel interface for analog radio and video capture and output drivers
In user space
Multimedia middleware
Gstreamer- Pipeline based multimedia framework written in the C programming
language with the type system based on GObject
OpenMAX- Cross-platform set of C-language programming interfaces that provides
abstractions for routines especially useful for audio, video, and still images
18
Is V4L2 enough for mobile
digital camera modules?
No way
19
Framework for digital camera
20
Camera subsystem in Linux kernel aspect
Camera interfaceCamera interface
Buffer handlingBuffer
handlingResolution handling
Resolution handling
Data format handling
(YUV/RAW/JPEG…)
Data format handling
(YUV/RAW/JPEG…)
CommandinterfaceCommandinterface
V4L2 driverV4L2 driver
Camera module
Exposure
control
Exposure
control
Lenscontro
l
Lenscontro
l
Strobecontro
l
Strobecontro
l
White balance control
White balance control
Effect contro
l
Effect contro
l
FacedetectFace
detect
Zoomcontro
l
Zoomcontro
l
.
.
.
.
.
.
V4L2 driverV4L2 driver
videobufvideobuf-core
videobuf-sg-dma
videodevV4l2-dev
V4l2-ioctl
V4l2-subdev(new)
V4l2-device
V4l2-int-
device(old)
V4L2
21
How does it work with V4L2?
Open deviceOpen
device
VIDIOC_QUERYCAP
VIDIOC_QUERYCAP
Application or Middleware
VIDIOC_S_INPUT
VIDIOC_S_INPUT
VIDIOC_REQBUF
VIDIOC_REQBUF
VIDIOC_QBUF
VIDIOC_QBUF
VIDIOC_STREAMON
VIDIOC_STREAMON
VIDIOC_QBUFVIDIOC_QBUF VIDIOC_DQBUF
VIDIOC_DQBUF
VIDIOC_QUERYBUF
VIDIOC_QUERYBUF
Copy memory to userspace
22
New V4L2 APIs for digital camera
Exposure control
23
Need for detailed control in exposure- We’ve got camera devices with mechanical shutter
- Iris could be handled also
- ISO controlled output is somehow useful
Why not using enhanced exposure control like a regular digital camera?
24
What we’ve got in mainline V4L2 right now?
#define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1)enum v4l2_exposure_auto_type { V4L2_EXPOSURE_AUTO = 0, V4L2_EXPOSURE_MANUAL = 1, V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3};#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
No way to control iris
No way to control exposure metering method
25
Additional exposure control
enum v4l2_exposure_auto_type {V4L2_EXPOSURE_AUTO = 0,V4L2_EXPOSURE_MANUAL = 1,V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,
V4L2_EXPOSURE_APERTURE_PRIORITY = 3,+ /* Additional features for digital camera */+ V4L2_EXPOSURE_ISO_PRIORITY =4,};#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
. . .
+#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)
26
Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products
+#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19)+#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20)+#define V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE+21)
Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU
{ .id = V4L2_CID_CAM_APERTURE, .type = V4L2_CTRL_TYPE_MENU, .name = "Aperture", .minimum = 0, .maximum = 4, .step = 1, .default_value = 0,},
static const char *camera_iris_stages[] = {/* This module supports 5 Iris stages * on it's own but F number depends * how you package Lens module */
"2.8“, "4“, "5.6“, "8“, "11“, NULL};
27
New exposure metering control
+/* Exposure Methods */+#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17)+enum v4l2_photometry_mode {+ V4L2_PHOTOMETRY_MULTISEG = 0, + V4L2_PHOTOMETRY_CWA = 1,+ V4L2_PHOTOMETRY_SPOT = 2,+ V4L2_PHOTOMETRY_AFSPOT = 3, +};
28
New V4L2 APIs for digital camera
Lens control
29
Need for enhanced Focus mode API
- Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam
- High-end camera modules support for various focus mode presets like
•AF Macro
•AF-S / AF-C
For compatibilityFocus control API should be defined
30
Focus Mode & Enumeration values
#define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10)#define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11)+#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13)
+/* Focus Methods */+enum v4l2_focus_mode {+ V4L2_FOCUS_AUTO = 0,+ V4L2_FOCUS_MANUAL = 1,+ V4L2_FOCUS_MACRO = 2,+ V4L2_FOCUS_CONTINUOUS = 3,+};
31
New V4L2 APIs for digital camera
Object recognition
32
Object detection
Gee!It recognized my
face!Hello everyone
33
What do we set?- Detect mode
- Type of object
- How many objects to detect
- Triggered action
What do we get?- How many objects are detected
- Detected object’s type
- Detected object’s coordinates
34
New API for object recognition- New capability
#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)
- New IOCTL
35
+/* Object detection and triggered actions */+enum v4l2_recog_mode {+ V4L2_RECOGNITION_MODE_OFF = 0,+ V4L2_RECOGNITION_MODE_ON = 1,+ V4L2_RECOGNITION_MODE_LOCK = 2,+};++enum v4l2_recog_action {+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */+};++enum v4l2_recog_pattern {+ V4L2_RECOG_PATTERN_FACE = 0; /* Face */+ V4L2_RECOG_PATTERN_HUMAN = 1; /* Human */+ V4L2_RECOG_PATTERN_CHAR = 2; /* Character */+}
New API for object detection
36
+struct v4l2_recog_rect {+ enum v4l2_recog_pattern p; /* What pattern detected */+ struct v4l2_rect o; /* detected area */+ __u32 reserved[4];+}++struct v4l2_recognition_data {+ __u32 detect_cnt; /* detected object counter */+ struct v4l2_recog_rect obj; /* detected area */+ __u32 reserved[4];+};++struct v4l2_recognition {+ enum v4l2_recog_mode mode;+ enum v4l2_recog_pattern pattern; /* What pattern to detect */+ __u32 obj_num; /* How many object to detect */+ __u32 detect_idx; /* select detected object */+ struct v4l2_recog_data data; /* read only */+ enum v4l2_recognition_action action;+ __u32 reserved[4];+};
New API for object detection
37
What do we set?- Detect mode
- How many objects to detect
- Triggered action
What do we get?- How many objects are detected
- Detected object’s coordinates
38
New API for object recognition- New capability
#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)
- New IOCTL
39
+/* Object detection and triggered actions */+enum v4l2_recognition_mode {+ V4L2_RECOGNITION_MODE_OFF = 0,+ V4L2_RECOGNITION_MODE_ON = 1,+ V4L2_RECOGNITION_MODE_LOCK = 2,+};++enum v4l2_recognition_action {+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */+};++struct v4l2_recognition_data {+ __u8 detect_cnt; /* detected object counter */+ struct v4l2_rect o; /* detected area */+};++struct v4l2_recognition {+ enum v4l2_recognition_mode mode;+ __u8 obj_num; /* How many object to detect */+ __u8 detect_idx; /* select detected object */+ struct v4l2_recognition_data data; /* read only */+ enum v4l2_recognition_action action;+};
New API for object detection
40
New V4L2 APIs for digital camera
Dual camera
41
What is dual camera?
Dual camera applications- Video telephony
- Camera applications• Taking self portrait shots
Flip side camera: for still shots
Front side camera: for special purpose
42
H/W restrictions in Application Processor’s camera peripheral interface
Megacamera
VGAcamera
Datapath
43
H/W restrictions in Application Processor’s camera peripheral interface
“SINGLE” camera interfaceon Application Processor !“SINGLE” camera interfaceon Application Processor !
Only one camera could be using data pins at a time
Only one camera could be using data pins at a time
Each camera is different inEach camera is different in
Working resolution
(SYNC)
Working resolution
(SYNC)Working MCLKWorking MCLK Working PCLKWorking PCLK
Only one camera could be handled at a timeOnly one camera could be handled at a time
44
We already have API for dual camera- Enumerate video inputs
- Query or select the current video input
VIDIOC_ENUMINPUTVIDIOC_ENUMINPUT
VIDIOC_G_INPUTVIDIOC_G_INPUT
VIDIOC_S_INPUTVIDIOC_S_INPUT
45
What do we need to consider?
•Context of each camera module•Capability : Supported color spaces, resolutions, additional functions…•Latest working status : resolution, color space, effect and so on
•Context of each camera module•Capability : Supported color spaces, resolutions, additional functions…•Latest working status : resolution, color space, effect and so on
•Context of camera interface•Working information of the camera module getting switched : Clock information and so on
•Context of camera interface•Working information of the camera module getting switched : Clock information and so on
•Let user space application get noticed about camera input devices
•Don’t forget VIDIOC_ENUMINPUT•Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES
•Let user space application get noticed about camera input devices
•Don’t forget VIDIOC_ENUMINPUT•Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES
46
How to switch active camera
Condition 2Shared things
Camera power enable pinCamera data pinsMCLK
Condition 3Dedicated things
Camera enable pinsCamera reset pins
Condition 1How to enable camera module:
Give power sourceEnable expecting cameraGive MCLK CAM
POWERMCLK DATA PINS
MegaEnable
MegaReset
VGAEnable
VGAReset
Mega camera VGA camera
CPU
47
STREAMOFF
Camera (A)
initialize
Camera (A)
initialize
Start preview
Start preview
Switch to camera (B)
Camera (B)
initialize
Camera (B)
initialize
Camera interface reset
& setting
Start preview
Start preview
Reset camera interface’s
current setting
Make a proper MCLK for camera
(B)
Setup resolution,
color-space, DMA
Stop previewStop DMA
Cam power off
Camera (B) Enable pin on
Camera (A) enable pin off
Campower on
Stop previe
w
Stop previe
w
STREAMON
G_INPUT/
S_INPUT
G_INPUT/
S_INPUT
48
Porting issues in SoC camera subsystem
49
I2C protocol has start and stop condition
Some camera modules generate unexpected noise until their power up sequence- Makes I2C bus in bus busy state
50
How to prevent this malfunctioning issue
Use a “noise free on power up” camera module
Use a level shifter to block and ignore noise from camera module
Change functionality of I2C pins while turning on camera module
OR
OR
51
All about between camera interface and camera module- Expected resolution from camera interface
- Served resolution from camera module
- Symptoms depend on camera interface
Expecting resolution is bigger than served resolution- A “select time out” error in user space application or
- Preview image is floating in some direction
Serving resolution is bigger than expected resolution- A “select time out” error in user space application or
- Preview angle looks like to be zoomed in
52
Color order mismatched between camera interface and camera module
53
Preview freezing while checking lens response- Using while() loop while it checks response, preview will
freeze until focusing job gets finished
- If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing.
54
Solving preview freezing while checking lens statusstruct platform_isp {
const struct platform_data *pdata;struct v4l2_int_device *v4l2_int_device;struct i2c_client *i2c_client;int reschk;
. . . . . . . . .}
static int ioctl_s_ctrl(struct v4l2_int_device *s,struct v4l2_control *vc)
{ struct platform_isp *isp = s->priv; struct i2c_client *client = isp->i2c_client; switch (vc->id) { case V4L2_CID_FOCUS_AUTO: isp->vfm = vc->value; if (vc->value == V4L2_CID_FOCUS_MACRO) { err = i2c_command_for_isp( client, focus_macro ); isp->reschk = 1; } break; }}
If isp->reschk is Set, The kernel thread is active.
55
Solving preview freezing while checking lens statusint pseudo_reschk_thread (void *data){ int err = 0;……… do { switch (isp->reschk) { case 1: err = i2c_command_for_isp(client, focus_check); if (err < 0) { isp->reschk = 1; /* i2c command function failed. */ } else { isp->reschk = 0; /* if succeed, reschk ended. */ } break;…… } } while(1); return 0;}
This function return when the ISP ismatched the focus(+ return) or timeout(- return).
This is able to re-check to the next check time.
Platform Data Vairable for isp, or static variable.
56
JPEG has no synchronization protocol when it gets transferred
All about camera interface and camera module’s data synchronization
- When camera interface supports JPEG sync• Follow user manual and make camera module to follow that
protocol
- When camera interface does not support JPEG sync• Make synchronization protocol: make JPEG data to be
synchronized with VSYNC
57
Q&A
58
Thank you
Top Related