Using OpenCV in iOS with Swift

[This report is also available in English.]

These days you can run homemade iOS apps on a real device without being an Apple Developer member.

And apparently, OpenCV works in Xcode too.



So I decided to build a face detection app using OpenCV on a whim.
…but explaining face recognition would take too long, so let's start with a Sobel edge detection demo.

Sample code for this post is available on GitHub.

What you need

• Mac
  
  Whatever Mac you have lying around.
• iPhone or iPad
  
  Most households have a couple of these.
• Xcode
  
  The iOS development environment. Free for development. Impressive.
  
  Install from the App Store.
• OpenCV
  
  The industry standard for image processing.
  With the method below, version 3.0.0 gets installed.

Install CocoaPods

You can drop in the OpenCV library directly, but since CocoaPods is available as a package manager, let's use it.


Open Terminal (the one that looks like this):

A lot of dependencies get pulled in during installation, but as long as there are no errors it's fine:

iMac:~ $ sudo gem install cocoapods
Fetching: i18n-0.7.0gem (100%)
Successfully installed i18n-0.7.0
    : (omitted)
Parsing documentation for cocoapods-1.0.1
Installing ri documentation for cocoapods-1.0.1
23 gems installed
iMac:~ $ pod setup

Create a Project

In Xcode, each app's source code and assets live in a "Project." Since OpenCV is installed per-project, create the Project first.



Launch Xcode from the dock and select "Create a new Xcode Project":

Choose Single View Application for now:

Fill in the app name and other fields however you like:

Project created. Leave defaults for now — detailed settings are only needed when publishing to the App Store:

Installing OpenCV

Close the Xcode Project for now.

Back in Terminal, navigate to the directory containing the .xcodeproj, create a Podfile, and install OpenCV.

Set the minimum iOS version to 7.0 — targeting 6.x or lower leads to a swamp of issues.

iMac: ~ $ cd FaceRecogApp
iMac:FaceRecogApp $ vi Podfile
iMac:FaceRecogApp $ cat Podfile
target "FaceRecogApp" do
echo 'platform :ios, "7.0"
pod 'OpenCV' > Podfile
end
iMac:FaceRecogApp $ pod install

This takes a while, but once it completes without errors, OpenCV is installed.

Double-click the .xcworkspace file to reopen the project in Xcode:

Capturing video in the app

Most iOS Hello World tutorials start with "in Storyboard, drag a button..." — but if you've never touched iOS development before, jumping into Storyboard makes things more confusing, not less. Let's go straight to code.



Open ViewController.swift inside the project in Xcode and write the following.
This gives you a basic live camera preview app. See the comments for what each part does:

import UIKit
import AVFoundation // Camera video library

class ViewController: UIViewController {

var input:AVCaptureDeviceInput! // Video input

var cameraView:UIView! // View for preview display
var session:AVCaptureSession! // Session
var camera:AVCaptureDevice! // Camera device

override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view, typically from a nib.
}

override func didReceiveMemoryWarning() {
super.didReceiveMemoryWarning()
// Dispose of any resources that can be recreated.
}

// Screen initialization
override func viewWillAppear(animated: Bool) {

// Stretch preview to fill screen
let screenWidth = UIScreen.mainScreen().bounds.size.width;
let screenHeight = UIScreen.mainScreen().bounds.size.height;
cameraView = UIView(frame: CGRectMake(0.0, 0.0, screenWidth, screenHeight))

// Find rear camera
// Use AVCaptureDevicePosition.Front for selfie camera
session = AVCaptureSession()
for captureDevice: AnyObject in AVCaptureDevice.devices() {
if captureDevice.position == AVCaptureDevicePosition.Back {
camera = captureDevice as? AVCaptureDevice
break
}
}

// Get camera video
do {
input = try AVCaptureDeviceInput(device: camera) as AVCaptureDeviceInput
} catch let error as NSError {
print(error)
}

if( session.canAddInput(input)) {
session.addInput(input)
}

// Display on screen
let previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer.frame = cameraView.frame
previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill

self.view.layer.addSublayer(previewLayer)

session.startRunning()
}
}

Build, run, and fail

Since this uses the camera, the simulator won't work.

Connect your iPhone via USB, then press the run button in Xcode.



…it fails to build.

Xcode 7 uses an intermediate format called "bitcode," but OpenCV doesn't include bitcode, causing a linker error.



Select the project name in Xcode, go to Build Settings, and disable bitcode:

Try again and the app launches on iPhone this time.

Extracting individual frames for processing

As-is, the captured video is just displayed live. To process each frame, we need to extract it as a UIImage object:

// Inherit AVCaptureVideoDataOutputSampleBufferDelegate to handle frame buffers
class ViewController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {

var input:AVCaptureDeviceInput! // Video input
var output:AVCaptureVideoDataOutput! // Video output

Push captured frames into a queue and process them via a delegate.

Since image processing can be slow and fall behind, late frames are discarded.
Ideally you'd also reduce the frame rate, but I haven't done that here. The device may get warm.

// Screen initialization
override func viewWillAppear(animated: Bool) {

// Stretch preview to fill screen
let screenWidth = UIScreen.mainScreen().bounds.size.width;
let screenHeight = UIScreen.mainScreen().bounds.size.height;
cameraView = UIImageView(frame: CGRectMake(0.0, 0.0, screenWidth, screenHeight))

// Find rear camera
// Use AVCaptureDevicePosition.Front for selfie camera
session = AVCaptureSession()
for captureDevice: AnyObject in AVCaptureDevice.devices() {
if captureDevice.position == AVCaptureDevicePosition.Back {
camera = captureDevice as? AVCaptureDevice
break
}
}

// Get camera video
do {
input = try AVCaptureDeviceInput(device: camera) as AVCaptureDeviceInput
} catch let error as NSError {
print(error)
}

if( session.canAddInput(input)) {
session.addInput(input)
}

// Send camera video to image processing
output = AVCaptureVideoDataOutput()
output.videoSettings = [kCVPixelBufferPixelFormatTypeKey : Int(kCVPixelFormatType_32BGRA)]

// Delegate
let queue: dispatch_queue_t = dispatch_queue_create("videoqueue" , nil)
output.setSampleBufferDelegate(self, queue: queue)

// Drop late frames
output.alwaysDiscardsLateVideoFrames = true

// Add output to session
if session.canAddOutput(output) {
session.addOutput(output)
}

// Lock camera orientation
for connection in output.connections {
if let conn = connection as? AVCaptureConnection {
if conn.supportsVideoOrientation {
conn.videoOrientation = AVCaptureVideoOrientation.Portrait
}
}
}

// Display → output via captureOutput
self.view.addSubview(cameraView)

session.startRunning()
}

The delegate picks up the image buffer and creates a UIImage.

Honestly, I have no idea what half of this code does. I just ported a sample from Objective-C to Swift 2 without fully understanding it.



The resulting UIImage is passed to ImageProcessing.SobelFilter at the end for OpenCV processing:

// Update display
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) {

let image:UIImage = self.captureImage(sampleBuffer)

dispatch_async(dispatch_get_main_queue()) {
// Draw
self.cameraView.image = image
}
}

// Create UIImage from sampleBuffer
func captureImage(sampleBuffer:CMSampleBufferRef) -> UIImage {

// Get image
let imageBuffer: CVImageBufferRef = CMSampleBufferGetImageBuffer(sampleBuffer)!

// Lock base address
CVPixelBufferLockBaseAddress(imageBuffer, 0 )

// Image data info
let baseAddress: UnsafeMutablePointer<Void> = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0)

let bytesPerRow: Int = CVPixelBufferGetBytesPerRow(imageBuffer)
let width: Int = CVPixelBufferGetWidth(imageBuffer)
let height: Int = CVPixelBufferGetHeight(imageBuffer)
let bitmapInfo = CGImageAlphaInfo.PremultipliedFirst.rawValue|CGBitmapInfo.ByteOrder32Little.rawValue as UInt32

// RGB color space
let colorSpace: CGColorSpaceRef = CGColorSpaceCreateDeviceRGB()!
let newContext: CGContextRef = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, bitmapInfo)!
// Quartz Image
let imageRef: CGImageRef = CGBitmapContextCreateImage(newContext)!

// UIImage
let cameraImage: UIImage = UIImage(CGImage: imageRef)

// Apply Sobel filter via OpenCV
let resultImage: UIImage = ImageProcessing.SobelFilter(cameraImage)

return resultImage

}

Integrating OpenCV

OpenCV doesn't support Swift, and Swift can't call C++ directly — so you need a bridge.

Right-click the file list in Xcode, select "New File," and create a C++ file. Xcode will automatically create a header file alongside it.

When I created ImageProcessing.m, the header file turned out to be named {ProjectName}-Bridging-Header.h. Why?



Define the class in the header file:

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>

@interface ImageProcessing: NSObject
+(UIImage *)SobelFilter:(UIImage *)image;
@end

Xcode automatically gives the implementation file a .m extension, but C++ requires .mm.
Change the extension to .mm in the "Identity and Type" panel in the top right.



Then implement the method: convert UIImage to an OpenCV buffer, run the filter, and return a UIImage.

OpenCV makes this beautifully simple:

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>

#import "FaceRecogApp-Bridging-Header.h"

#import <opencv2/opencv.hpp>
#import <opencv2/imgcodecs/ios.h>

@implementation ImageProcessing: NSObject

+(UIImage *)SobelFilter:(UIImage *)image{

// Convert UIImage to cv::Mat
cv::Mat mat;
UIImageToMat(image, mat);

// Convert to grayscale
cv::Mat gray;
cv::cvtColor(mat, gray, cv::COLOR_BGR2GRAY);

// Edge detection
cv::Mat edge;
cv::Canny(gray, edge, 100, 200);

// Convert cv::Mat back to UIImage
UIImage *result = MatToUIImage(edge);
return result;
}
@end

Note: most online examples show #import <opencv2/highgui/ios.h>, but in OpenCV 3.0.0 this has moved to imgcodecs.



Also, many examples use cv::cvColor, but in 3.0.0 the correct spelling is cv::cvtColor.

Running it

Here's what it looks like running on an iPad:


Check on Amazon Check on Amazon