Droidcon NYC 2014: Building Custom Camera Applications

34
BUILDING CUSTOM CAMERA APPLICATIONS Droidcon NYC 2014 Presented by Huyen Tue Dao September 21, 2014 376dp wrap_content @+id/randomly_typing

description

Presentation slides from my Droidcon NYC 2014 talk on the Android Camera API and building custom camera applications with it.

Transcript of Droidcon NYC 2014: Building Custom Camera Applications

Page 1: Droidcon NYC 2014: Building Custom Camera Applications

BUILDING CUSTOM CAMERA APPLICATIONSDroidcon NYC 2014 Presented by Huyen Tue Dao September 21, 2014

376dp

wrap_content

@+id

/ran

doml

y_ty

ping

Page 2: Droidcon NYC 2014: Building Custom Camera Applications

Where to start

Camera setup

Image capture

Camera information

Camera parameters

Camera API in L

BUILDING A CUSTOM CAMERA APPLICATION

2

Page 3: Droidcon NYC 2014: Building Custom Camera Applications

Mobile developer: native Android and native iOS

Android developer 4 years (personally use one)

Computer Engineering, University of MD, College Park

Gamer (video, board, card, anything): mostly Dota 2, QuizUp

ABOUT ME

3© 2011 Adam C Beamish

Page 4: Droidcon NYC 2014: Building Custom Camera Applications

WHERE TO STARTWhat you want vs what you need

Balance having critical features with supporting

target user base

What features the API supportsdeciding minimum SDK

using Build.VERSION

What features the device camera has

using use-feature and PackageManager querying support level from the Camera

being aware of manufacturer/device quirks

WANT

CAN

You can’t always get what you want. But if you try sometimes well you just mind find You get what you need.

Page 5: Droidcon NYC 2014: Building Custom Camera Applications

<uses-permission android:name="android.permission.CAMERA" />

Without any <uses-feature> specifications, to install a device would require:

a back-facing camera

a front-facing camera

auto-focus

flash

From docs: “This will automatically enforce the <uses-feature> manifest element for all camera features.”

If any of the above are not requirements, specify <uses-feature>: <uses-feature android:name="android.hardware.camera.autofocus" android:required="false"/>

CAMERA SETUP: THE MANIFEST

5

Page 6: Droidcon NYC 2014: Building Custom Camera Applications

CAMERA SETUP: CAMERA + PREVIEWThe order of things…

Add a SurfaceView to your layout for the camera preview

Implement a SurfaceHolder.Callback: surfaceCreated(), surfaceChanged(),

surfaceDestroyed()

Pass the callback to the SurfaceHolder instance of the SurfaceView

Get android.hardware.Camera instance via Camera.open()

Call Camera.getParameters() and perform any initial setup

Call Camera.startPreview()

wait for surface to be created: #surfaceCreated

Call Camera.setPreviewDisplay() with the SurfaceHolder

Page 7: Droidcon NYC 2014: Building Custom Camera Applications

CAMERA SETUP: CAMERA + PREVIEWOpen/release camera

Page 8: Droidcon NYC 2014: Building Custom Camera Applications

CAMERA SETUP: CAMERA + PREVIEWImplementing SurfaceHolder.Callback

Page 9: Droidcon NYC 2014: Building Custom Camera Applications

“Did you speak the exact words?”

Must call Camera.setPreviewDisplay() after the surface is created

Otherwise, no error, no preview

Preview size changes must be made between Camera.stopPreview()/Camera.startPreview()

all Camera.release() in onPause() to free up the camera

The surface is destroyed when SurfaceView is View.INVISIBLE.

CAMERA SETUP: THINGS TO NOTE

9

Page 10: Droidcon NYC 2014: Building Custom Camera Applications

Call Camera.takePicture(ShutterCallback, PictureCallback, PictureCallback, PictureCallback)

The 3 PictureCallback parameters = 3 image formats raw: uncompressed postview: scaled, fully processed JPEG: compressed Raw and postview support depends on device

ShutterCallback (more or less) invoked the moment that camera sensor captures an image

Camera.takePicture() stops the camera preview; call Camera.startPreview() in/after picture taking callbacks

IMAGE CAPTURE

10

Page 11: Droidcon NYC 2014: Building Custom Camera Applications

IMAGE CAPTUREwith a JPEG callback

Page 12: Droidcon NYC 2014: Building Custom Camera Applications

CameraInfo: per device camera orientation: angle of rotation when facing the camera for the image to match the natural orientation of the device facing: front or back whether the shutter sound can be disabled

Camera.getCameraInfo(): camera IDs are 0-based indices

Use CameraInfo to swap between front and back Use PackageManager to check if a front camera exists if front camera is not required by <use-feature> Close the current camera before swapping

CAMERA INFO: SWITCHING IT UP

12

Page 13: Droidcon NYC 2014: Building Custom Camera Applications

CAMERA INFOFinding the right camera

Page 14: Droidcon NYC 2014: Building Custom Camera Applications

14

Page 15: Droidcon NYC 2014: Building Custom Camera Applications

15

Page 16: Droidcon NYC 2014: Building Custom Camera Applications

DEVICE ORIENTATION VS CAMERA ORIENTATIONi.e. your camera is sideways

Natural camera orientation Natural device orientation

Camera.Info.orientation = 90

Camera.Info.orientation = 270

90°

270°

(looking at the device from the front)

Page 17: Droidcon NYC 2014: Building Custom Camera Applications

17

Page 18: Droidcon NYC 2014: Building Custom Camera Applications

Camera/display rotation + configuration changes -> sucks

Complicated

Device and camera orientation changes don’t play nice

Option #1: change activity orientation animation (API 18+): WindowManager.LayoutParams.html#rotationAnimation

Option #2: lock the activity orientation

CAMERA ROTATION

18

Page 19: Droidcon NYC 2014: Building Custom Camera Applications

Camera.Parameters information about and access to features

Device support:

PackageManager to check for auto-focus and flash

Otherwise check getter return values in Camera.Parameters

!isAutoWhiteBalanceLockSupported()

Or returns 0 or null

API level also a factor (of course)

CAMERA PARAMETERS

19

Page 20: Droidcon NYC 2014: Building Custom Camera Applications

CAMERA.PARAMETERS SETTING

Page 21: Droidcon NYC 2014: Building Custom Camera Applications

Camera.Parameters changes happen immediately (mostly)

Saving/restoring parameters Camera.Parameters.flatten() Camera.Parameters.unflatten()

Change values on a Camera.Parameters instance and pass it to Camera.setParameters()

Always call Camera.getParameters()

Do not hold onto Camera.Parameters instances

Only flattened string makes it past Camera service

CAMERA SETTINGS: UPDATING

21

Page 22: Droidcon NYC 2014: Building Custom Camera Applications

22

Page 23: Droidcon NYC 2014: Building Custom Camera Applications

Camera.Area rectangular bounds within viewfinder

metering areas: measuring light to calculate exposure + WB

focus areas: prioritizing focus

Camera.Parameters.setMeteringAreas(List<Camera.Area>)

Camera.Parameters.setFocusAreas(List<Camera.Area>)

Relative to the current zoom: cannot be outside FOV

Camera viewfinder/sensor has its own coordinate system

CAMERA FEATURES: AREAS

23

Page 24: Droidcon NYC 2014: Building Custom Camera Applications

VIEWFINDER VS VIEW COORDINATESTranslating between what the camera sees and what the user sees

Camera

(0,0)

H

W

View

1000-1000

1000

-1000

Viewfinder

Full FOV

Viewfinder

zoom = 1.5x

preview surface in layout

Page 25: Droidcon NYC 2014: Building Custom Camera Applications

25

Page 26: Droidcon NYC 2014: Building Custom Camera Applications

Camera.Parameters.getMaxNumDetectedFaces()

Camera

FaceDetectionListener, setFaceDetectionListener()

startFaceDetection(), stopFaceDetection()

Camera.Face: data object ID: unique per face face bounds score: how confident are we that this is a face? eye and face coordinates

FACE DETECTION

26

Page 27: Droidcon NYC 2014: Building Custom Camera Applications

FaceDetectionListener.onFaceDetection() is called frequently

Start only when actually needed

Avoid intensive work inside the callback

No event when no faces are detected

Modifying the UI in response to face detection: timer or postDelayed to reset UI after some time without detection event

FACE DETECTION

27

Page 28: Droidcon NYC 2014: Building Custom Camera Applications

Driven by a new Hardware Abstract Layer (HAL3)

HAL Version 1: black box: 3 modes (preview, still, video)

HAL3: all requests handled same way, “unified view”

More user control for both capture and post-processing

Maintainability and efficiency

CAMERA API 2 IN L: THE FUTURE IS AWESOME

28

Page 29: Droidcon NYC 2014: Building Custom Camera Applications

HAL V1 VS HAL V3https://source.android.com/devices/camera/camera3.html

Page 30: Droidcon NYC 2014: Building Custom Camera Applications

HAL3 (EXPANDED)https://source.android.com/devices/camera/camera3_requests_hal.html

Page 31: Droidcon NYC 2014: Building Custom Camera Applications

More metadata

Camera information

Capture configuration part of request and result

Dynamic metadata: timestamps, exposure time

More output options

Multiple Surface instances receive results from a single request

App-visible JPEG, YUV, RAW Bayer buffers

CAMERA API 2 IN L: THE FUTURE IS AWESOME

31

Page 32: Droidcon NYC 2014: Building Custom Camera Applications

Camera (static bits) -> CameraManager

AvailabilityListener: removable cameras

Asynchronous opening + CameraDevice.StateListener

Camera (function access) -> CameraDevice

Actually get the ID of the camera!

CameraCaptureSession/CaptureRequest

Parameters -> CaptureRequest.Builder.set() to configure

CameraInfo -> CameraCharacteristics

CameraMetadata parent of CameraCharacteristics, CaptureRequest, CaptureResult, TotalCaptureResult

CAMERA API 2 IN L: THE FUTURE IS AWESOME

32

Page 33: Droidcon NYC 2014: Building Custom Camera Applications

Lessons from API 1 do not go away (remember the words)

More complex… but more is still better More control and performance More consistency in the API More for developers to leverage

It’s going to be…

CAMERA API 2 IN L: THE FUTURE IS AWESOME

33

Page 34: Droidcon NYC 2014: Building Custom Camera Applications

THANK YOU + QUESTIONS?

Huyen Tue Dao Lead developer | Owner Randomly Typing !@queencodemonkey [email protected] randomlytyping.com !speakerdeck.com/randomlytyping slideshare.net/randomlytyping

34

Code will coming to github !

Things To check out !

Building Apps with Multimedia: Capturing Photos developer.android.com/training/camera/

index.html !

Android Design in Action: Camera Apps youtube.com/watch?v=OLSa7fErTAM

!Standford Digital Image Processing Class

stanford.edu/class/ee368/Android/index.html !

DevBytes: Android L - Camera2 API youtube.com/watch?v=Xtp3tH27OFs

!Sample Camera 2 API app:

github.com/googlesamples/android-Camera2Basic/blob/master/Camera2BasicSample

!Digital Camera Sensor Sizes

gizmag.com/camera-sensor-size-guide/26684/