Pan Tilt #4: Face tracking

This is post #4 in a series about making a pan tilt controller for the Raspberry Pi and it's camera module. Post 1 discussed the Lego build. Post 2 mounted the camera, Post 3 used colour tracking to control the servos.

Tracking coloured balls is all well and good but what would be really neat would be tracking a face with the camera. OpenCV comes out of the box with face detection so I got to work.

The joystick and colour tracking controllers from the previous posts laid the ground work to make a pluggable architecture for my controllers. It didn't take long to add face detection capabilities. The source is on github.

After getting it working there are some problems with face tracking however. So much so I haven't bothered to record a video of it.

Problem: Raspberry Pi B+ Performance

Running on my Intel Core i7, face detection takes a few milliseconds. Running on the B+ Pi however it can take 250ms @ 128x96 per frame with the default settings on the cascade classifier. The face detection - or any object detection for that matter - must be completed as fast as possible to allow the camera to continually move and track the target. Processing times of 250ms per frame is too slow.  The timing can be improved by tuning the inputs to the classifier - in this case I increased the scaling from 1.1 to 1.4 at the expense of more false negatives but even at 1.4 there isn't enough performance improvement to save it.

Below are the rough the face detection response times I measured.

UPDATE June 2015 - Now with Pi 2 timings, what a difference! Note there is a floor of 400ms/10 frames with TBB=false. The floor is because I didn't realise that the classifier was eating up the default 25 fps. When I made the TBB=true measurements, I increased the capture framerate sufficiently to ensure the classifier wasn't starved of work. So read tbb=false measurements of 400 as '400 or even faster'. At 320x240 we can see the benefits of TBB kicking in.

UPDATE Nov 2016 - Pi 3 and i7 timings added. 350m seems a suspect floor for the i7 - there must be a bottleneck somewhere

Resolution	Framerate / 10 frames (ms)
	Raspberry Pi B+		Raspberry Pi 2		Raspberry Pi 3	Intel i7-4770
	Scale 1.1	Scale 1.4	Scale 1.1, TBB=false	Scale 1.1, TBB=true	Scale 1.1, TBB=true	Scale 1.1, TBB=true
64x48	1480 (min size=5)	undetected	396	170-200	-
128x96	3250 (min size=20)	2100	400	330	200-220	350
160x120	4880	2900	400-550	400-440	350-430	350
320x240	-	-	1800	1300	850-1100	600-800 (minSize=20) 350 (minSize=50)

Problem: Full Frontal Views Required

Face tracking works well as long as the subject keeps their face pointed toward the camera. If they approach side on or turn their head to leave, the face detection drops out as it isn't trained for profile views. This makes the face tracking not quite as magical as originally envisaged.

Problem: Field of View

The stock lens for the Pi has a relatively narrow field of view. It doesn't take much deviation for the subject to move out of view of the camera and for it to come to a standstill. This is compounded by the face processing time required.

Conclusion

More work required!