After building and testing my Lego pan tilt mechanism and controlling it with a joystick (post 1 in this series), it was time to mount the Raspberry Pi Camera module on to it.
With the 25cm flat ribbon cable from eBay, mounting the camera was easy. The next obvious step was to use OpenCV to identify an object of interest in the camera frame before writing code to slew the servos such that the object came into the center of the frame. Repeat this fast enough and the camera will jump around tracking a selected target. Handily, from my marble sorting contraption, I had most of the code required to make my pan tilt camera a colour tracking machine. Given a red marble say, I could use OpenCV to track the region of colour, calculate its center and then command the servos to point the camera directly at the target.
Identifying the location of an object in the frame is all well and good however with that positional information in pixels, I needed to translate it into a servo movement that would move the camera so as to bring the object into the center of the view. Briefly I mistakenly thought I would need some form of PID control before I realised that the PID control is already baked into the servo's PWM circuitry. I just need to feed the servo desired a position and it will do its very best to slew directly to and maintain that position.
Hand tuned Calibration
First I tried my hand at obtaining the calibration data by hand. I expected the mapping of pixel deviation to servo movement to be linear, so one should be able to measure a few sample data points and use Linear Regression. The result of the regression would be a formula to calculate the required servo movement for any pixel deviation.
I added some controls to my PiCamCV WinForms project to facilitate making the measurements - painting two reticles. One in the center and another at any arbitrary point.
The process was a bit tedious but I managed to get 10 samples for each axis and the resulting graphs were nicely linear - as expected.
In the above graphs, the x-axis represents pixel deviation from the center. The y-axis is the servo movement required to bring the object into the center of the frame.
Hence observing the first the pan regression graph, the required servo movement (y) for an object that is x pixels from the center is given by:
y = 0.0933x + 0.0864
The regression analysis gives R of 0.9991 meaning the formula is a very good fit to the data. R=1 means an exact match. Note however that the hand calibration was done at a capture resolution of 320x240 so capturing frames at any other resolution requires scaling. Reducing capture resolution is desirable to increase the processing speed - less pixels, less compute time.
After taking these results and coding them up, it was exciting to see the camera tracking a red ball. It was far from perfect though - it acted fairly erratically rather than the hoped for behaviour of zeroing in on a target with laser like precision. I'll discuss why later in this post but initially I thought my hand measurements were out. I decided I should be able to automate the calibration and get a much larger and accurate set of sample data across the main resolutions I wanted to capture at.
After some consideration I realised I could improve the accuracy of my linear regression by automating the data samples. I coded up a controller that would make small servo deviations before measuring how far the center of a red marble moved and therefore build up an accurate servo to pixel map. When tracking, reversing this map gives a ready made lookup table for every pixel deviation. Since the camera is mounted on the pan/tilt the distance to target doesn't come into play - it would however if the camera was fixed independently of the pan tilt platform.
Here is the calibration in action:
Note the regression formula differs to my hand measured one due to the scaling between the respective capture resolutions of 320x240 for hand calibration vs 160x120 auto calibrated.
Eratic Behaviour Solved
In the process of building and debugging my auto calibrating engine, I realised there were two factors at play in the erratic behaviour seen earlier and it wasn't necessarily my grossly inaccurate hand measured regression.
One, the images being pulled out of the buffer were stale. After slewing the servo and then pulling out a frame I could see the image being used for the measurement was completely wrong - it was from where the servo had been a second before, not where it was now. After some sleuthing I realised I had to discard 2 frames before making a measurement on pixel deviation.
Secondly the auto calibration wasn't always waiting long enough for the servo to complete its movement and was capturing images with motion blur before the camera had reached its final resting position. These two factors were hampering the tracking solution and once identified and solved, improved the accuracy greatly.