Sri Malireddi

Computer Vision in iOS - Core Camera Pipeline

November 05, 2019

Computer Vision on mobile is fun! Here are a few reasons why I personally love computer vision on mobile when compared to traditional desktop based systems.

  • Everyone has a smartphone with decent camera now-a-days. So, you need not have to buy a webcam or high resolution camera and connect it to computer through USB cable.
  • Portability - Designing computer vision apps targeted for mobile phones are trivially scalable. Imagine yourselves carrying your workstation along with webcam everywhere you want to present your demo vs. just grabbing your phone from pocket and showcasing it.
  • Processing Power - The present day smart phones are as powerful as a computer. And year after year their processing power is still increasing. Integrating high performance embedded vision applications into smart phones is way easier than Raspberry Pi (or) Arduino.
  • Scalability - Developing computer vision algorithms tailored for mobile phones can easily lead to production when compared to desktop development.

Smart phones are fully equipped with CPU, GPU, DSPs and (recently) custom Neural chips which can be utilised based on the application requirements. The only disadvantage with mobile computer vision is that you can’t take an already existing real-time desktop application and expect it to work real-time on smart phones. Optimisation plays a key role in mobile computer vision. Mobile battery is limited, hence energy usage of your algorithm matters! If you are designing a heavy CV based system, you can’t schedule the whole operations on CPU. You might need to come with some algorithm and system strategies that can reduce the CPU usage.

Enough discussion, let’s dive into the topic of the blog. In this blog, I will introduce you to a very easy way to integrate camera into your application with as minimal code as possible.

TLDR; The 4 magic lines you need to add to your ViewController to easily integrate the camera into your application are as follows:

import CVLib
@IBOutlet var cameraView: CameraView!
self.cameraView.start()
self.cameraView.stop()

If you are an iOS expert, you can integrate CVLib pods and use the above 4 lines in your ViewController to fire up the camera. For beginners, you can continue reading further for step-by-step tutorial.

Steps to integrate camera

Step 1 - Install CVLib Cocoapod

  • Without wasting any more time create a new Single View Application with your desired product name and set the language as Swift.
  • Open the location of your project in Terminal and type the following.
$ pod init
  • This should create a Podfile in your project directory. Now open this Podfile in the editor of your choice. I generally use VS Code. Add the following edits to your Podfile.
# Uncomment the next line to define a global platform for your project
platform :ios, '10.0'
source 'https://github.com/Cocoapods/Specs.git'
target 'TestCameraView' do
  # Comment the next line if you don't want to use dynamic framework
  use_frameworks!

  # Pods for TestCameraView
  pod 'CVLib'end
  • Now go back to terminal and type -
$ pod install
$ pod update
  • This should install the latest version of CVLib into your Xcode project workspace. Now follow the instructions in your terminal and open .xcworkspace file of your project.

Step 2 - Integrate Camera View

  • In Info.plist make sure to add the permissions for camera.

Info.plist

  • In Main.storyboard, select View from the editor.

Main.storyboard

  • From Identity Inspector’s Class, select CameraView from the drop-down Menu. You should now be seeing the View renamed to Camera View.

CameraView

  • Setup an outlet for CameraView in ViewController.Swift.
  • Now simply add the following “4 lines of code” in your ViewController to start and stop the camera session.
import UIKit
// MARK: Step 1 - import the CVLibimport CVLib
class ViewController: UIViewController {

    // MARK: Step 2 - initialize CameraView    @IBOutlet var cameraView: CameraView!    
    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view.
    }
    
    // Start session
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        // MARK: Step 3 - Start Camera        self.cameraView.start()    }
    
    // Stop session
    override func viewWillDisappear(_ animated: Bool) {
        // MARK: Step 4 - Stop Camera        self.cameraView.stop()        super.viewWillDisappear(animated)
    }


}
  • Now simply build and run the application and you should be seeing the following Camera View in your application.

Screenshot