Pan Tilt #4: Face tracking

May 31, 2015

This is post #4 in a series about making a pan tilt controller for the Raspberry Pi and it's camera module. Post 1 discussed the Lego build. Post 2 mounted the camera, Post 3 used colour tracking to control the servos.

Tracking coloured balls is all well and good but what would be really neat would be tracking a face with the camera. OpenCV comes out of the box with face detection so I got to work.

The joystick and colour tracking controllers from the previous posts laid the ground work to make a pluggable architecture for my controllers. It didn't take long to add face detection capabilities. The source is on github.

After getting it working there are some problems with face tracking however. So much so I haven't bothered to record a video of it.

Problem: Raspberry Pi B+ Performance

Running on my Intel Core i7, face detection takes a few milliseconds. Running on the B+ Pi however it can take 250ms @ 128x96 per frame with the default settings on the cascade classifier. The face detection - or any object detection for that matter - must be completed as fast as possible to allow the camera to continually move and track the target. Processing times of 250ms per frame is too slow.  The timing can be improved by tuning the inputs to the classifier - in this case I increased the scaling from 1.1 to 1.4 at the expense of more false negatives but even at 1.4 there isn't enough performance improvement to save it.

Below are the rough the face detection response times I measured.

Resolution Framerate / 10 frames (ms)
Raspberry Pi B+ Raspberry Pi 2 Raspberry Pi 3 Intel i7-4770
Scale 1.1 Scale 1.4 Scale 1.1, TBB=false Scale 1.1, TBB=true Scale 1.1, TBB=true Scale 1.1, TBB=true
64x48 1480 (min size=5) undetected 396 170-200 -
128x96 3250 (min size=20) 2100 400 330 200-220 350
160x120  4880  2900 400-550 400-440 350-430 350
320x240  -  - 1800 1300 850-1100 600-800 (minSize=20)
350 (minSize=50)

Problem: Full Frontal Views Required

Face tracking works well as long as the subject keeps their face pointed toward the camera. If they approach side on or turn their head to leave, the face detection drops out as it isn't trained for profile views. This makes the face tracking not quite as magical as originally envisaged.

Problem: Field of View

The stock lens for the Pi has a relatively narrow field of view. It doesn't take much deviation for the subject to move out of view of the camera and for it to come to a standstill. This is compounded by the face processing time required.


More work required!