Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai...
Transcript of Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai...
![Page 1: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/1.jpg)
Learning Core ImageGuanshan Liu@guanshanliu
CocoaHeads Shanghai Meetup in Dec. 1
![Page 2: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/2.jpg)
Who Am I
— Working on TTPod for iOS at Alibaba Inc.
— I love design and make apps
— Twitter: @guanshanliu
— Email: [email protected]
CocoaHeads Shanghai Meetup in Dec. 2
![Page 3: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/3.jpg)
What is Core Image
Core Image is a powerful image processing framework that allow you to easily add effects to still images and live video. It is built on top of OpenGL.
— It uses GPU to process image or CPU (kCIContextUseSoftwareRenderer: YES)
— Introduced in OS X 10.4, iOS 5
— You can create custom image kernels in iOS 8
CocoaHeads Shanghai Meetup in Dec. 3
![Page 4: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/4.jpg)
Overview
— CIContext It's where all image processing happens. Similar to CoreGraphics or OpenGL context.
— CIImage An image abstraction.
— CIFilter A filter takes one or more images as input, produces a CIImage object as output based on key-value pairs of input parameters.
CocoaHeads Shanghai Meetup in Dec. 4
![Page 5: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/5.jpg)
CIContext
In iOS 7, the CPU renderer was used when- GPU texture limits were exceeded- The application needed to render briefly in the background- The application wanted to render in a low priority thread
Copied from Session 514, WWDC 2014
CocoaHeads Shanghai Meetup in Dec. 5
![Page 6: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/6.jpg)
CIContext
Full support for images greater than the GPU limits in iOS 8- Input images can be > 4K- Output renders can be > 4K
GPU texture limits were exceeded - No longer a limit in iOS 8 Core Image
Copied from Session 514, WWDC 2014CocoaHeads Shanghai Meetup in Dec. 6
![Page 7: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/7.jpg)
CIContext
In iOS 8Renders within a short time of switching to background - Use faster GPU renderer - Serviced with a lower priority - Will not disturb foreground GPU usage
Copied from Session 514, WWDC 2014
CocoaHeads Shanghai Meetup in Dec. 7
![Page 8: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/8.jpg)
CIContext
— The application needed to render briefly in the background
— The application wanted to render in a low priority thread
— Can now request kCIContextPriorityRequestLow in iOS 8 Core Image
Copied from Session 514, WWDC 2014CocoaHeads Shanghai Meetup in Dec. 8
![Page 9: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/9.jpg)
CIImage
It can be created in many ways:
— Raw pixel data: NSData, CVPixelBufferRef, etc.
— Image data classes: UIImage, CGImageRef, etc.
— OpenGL textures
CocoaHeads Shanghai Meetup in Dec. 9
![Page 10: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/10.jpg)
CIFilterBuiltin Filters
— In Objective-C
[CIFilter filterNamesInCategory:kCICategoryBuiltIn]
— In Swift
CIFilter.filterNamesInCategory(kCICategoryBuiltIn)
CocoaHeads Shanghai Meetup in Dec. 10
![Page 11: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/11.jpg)
CIFilterBuiltin Filters
— 169 filters on OS X 10.10
— 127 filters on iOS 8
CocoaHeads Shanghai Meetup in Dec. 11
![Page 12: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/12.jpg)
CIFilter
Each filter has a dictionary containing filter's name, the kinds of input parameters the filters takes, the default and acceptable values, and its category.
CocoaHeads Shanghai Meetup in Dec. 12
![Page 13: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/13.jpg)
CIFilter
In Objective-C
NSArray *filters = [CIFilter filterNamesInCategory:kCICategoryBuiltIn]; for (NSString *filterName in filters) { CIFilter *filter = [CIFilter filterWithName:filterName]; NSLog(@"%@", [filter attributes]); }
In Swift
let filterNames = CIFilter.filterNamesInCategory(kCICategoryBuiltIn) as [String]for filterName in filterNames { let filter = CIFilter(name: filterName) println(filter.attributes())}
CocoaHeads Shanghai Meetup in Dec. 13
![Page 14: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/14.jpg)
Example - CISepiaTone
CocoaHeads Shanghai Meetup in Dec. 14
![Page 15: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/15.jpg)
Example - CISepiaTone
// Create a CIContextlet context = CIContext()
// Get CIImage from UIImagelet image = UIImage(named: "Image")!let input = CIImage(image: image)
// Create a fitlerlet filter = CIFilter(name: "CISepiaTone")filter.setValue(input, forKey: kCIInputImageKey)filter.setValue(1.0, forKey: kCIInputIntensityKey)
// Get output CIImage from the filterlet output = filter.outputImagelet extent = output.extent()
// Get UIImage from CIContextlet imageRef = context.createCGImage(output, fromRect: extent)let outputImage = UIImage(CGImage: imageRef, scale: image.scale, orientation: image.imageOrientation)!
CocoaHeads Shanghai Meetup in Dec. 15
![Page 16: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/16.jpg)
Demo
Sepia Tone Filter
CocoaHeads Shanghai Meetup in Dec. 16
![Page 17: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/17.jpg)
Example - Filter Chain
Filters can be chained together. It's like a pineline. Just put the output image of a filter as input image of the next filter.
CocoaHeads Shanghai Meetup in Dec. 17
![Page 18: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/18.jpg)
Auto-Enhancement
CIImage has a method autoAdjustmentFilters that returns an array of filters including red eye reduction, flesh tone, etc.
You can use the array to apply a filter chain to an image.
CocoaHeads Shanghai Meetup in Dec. 18
![Page 19: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/19.jpg)
Example - Filter Chain
func autoAdjustment(image: CIImage) -> CIImage { let filters = image.autoAdjustmentFilters() as [CIFilter] let output = filters.reduce(image, combine: { (input, filter) -> CIImage in filter.setValue(input, forKey: kCIInputImageKey) return filter.outputImage }) return output}
CocoaHeads Shanghai Meetup in Dec. 19
![Page 20: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/20.jpg)
Demo
Filter Chain
CocoaHeads Shanghai Meetup in Dec. 20
![Page 21: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/21.jpg)
Two More Demos
CocoaHeads Shanghai Meetup in Dec. 21
![Page 22: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/22.jpg)
Example - Custom Image Kernel
1. Subclass CIFilter
2. let kernel = CIKernel(string: kernelSource)
3. override var outputImage: CIImage { get } method using kernel.applyWithExtent
CocoaHeads Shanghai Meetup in Dec. 22
![Page 23: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/23.jpg)
Demo
Custom Image Kernel
CocoaHeads Shanghai Meetup in Dec. 23
![Page 24: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/24.jpg)
Example - Live Video Filter
glContext = EAGLContext(API: .OpenGLES3)glView.context = glContextcoreImageContext = CIContext(EAGLContext: glContext)
let videoOutput = AVCaptureVideoDataOutput()videoOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_32BGRA]videoOutput.setSampleBufferDelegate(self, queue: sessionQueue)session.addOutput(videoOutput)
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) { let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) var image = CIImage(CVPixelBuffer: pixelBuffer) image = sepiaTone(image) coreImageContext.drawImage(image, inRect: bounds, fromRect: bounds) glContext.presentRenderbuffer(Int(GL_RENDERBUFFER))}
CocoaHeads Shanghai Meetup in Dec. 24
![Page 25: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/25.jpg)
Demo
Live Video Filter
CocoaHeads Shanghai Meetup in Dec. 25
![Page 26: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/26.jpg)
Core Image with Functional Programming
typealias Filter = CIImage -> CIImage
func blur(radius: Double) -> Filter { return { image in let parameters = [ kCIInputRadiusKey: radius, kCIInputImageKey: image ] let filter = CIFilter(name: "CIGaussianBlur", withInputParameters: parameters) return filter.outputImage }}
More in Functional Programming in Swift
CocoaHeads Shanghai Meetup in Dec. 26
![Page 27: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/27.jpg)
Core Image with Functional Programming
func sepiaTone(intensity: Double) -> Filter { return { image in let parameters = [ kCIInputImageKey: image, kCIInputIntensityKey: intensity ] let filter = CIFilter(name: "CISepiaTone", withInputParameters: parameters) return filter.outputImage }}
More in Functional Programming in Swift
CocoaHeads Shanghai Meetup in Dec. 27
![Page 28: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/28.jpg)
Core Image with Functional Programming
infix operator ⋅ { associativity left }
public func ⋅ <T, U, V> (g: U -> V, f: T -> U) -> T -> V { return { x in g(f(x)) }}
let myFilter = sepiaTone(0.8) ⋅ blur(5)
More in Functional Programming in Swift
CocoaHeads Shanghai Meetup in Dec. 28
![Page 29: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/29.jpg)
ResourcesCore Image
— WWDC sessions
1. 2011: 129, 422
2. 2012: 510, 511
3. 2013: 509
4. 2014: 514, 515
— Beginning Core Image in iOS 6CocoaHeads Shanghai Meetup in Dec. 29
![Page 30: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/30.jpg)
Custom Image Kernel
— GPUImage by Brad Larson
Slides and sample codes of this talk
— Available on GitHub
CocoaHeads Shanghai Meetup in Dec. 30
![Page 31: Guanshan Liu - Meetupfiles.meetup.com/13201442/Learning Core Image.pdf · CocoaHeads Shanghai Meetup in Dec. 1. Who Am I — Working on TTPod for iOS at Alibaba Inc. — I love design](https://reader033.fdocuments.us/reader033/viewer/2022050604/5fab50bdfd038a5ef76aee17/html5/thumbnails/31.jpg)
Thank you!
CocoaHeads Shanghai Meetup in Dec. 31