We find that the best place to field test ultrawide-baseline stereo vision technology is at the pool on a sunny day!
Don’t tell my investors, but I am going to reveal how we build ultrawide-baseline stereo-vision cameras at NODAR…
This newsletter is about how machines see in 3D and what it costs, written for the people who decide what sensors ship. One technical idea per issue, no pitch.
Here is the first one: how we built a custom rig from scratch, start to finish.
We got a request to build a sensor to reconstruct the water surface at a 50-ish-meter range for an outdoor water-monitoring project. “Ok, that’s never been done before,” I thought when we first talked about the project. Then, the customer said, "We don’t just want that capability during the day, we also want to measure the water surface at NIGHT.” Um, ok, customer requests are always reasonable 🙂. Should be easier than the automotive companies asking us to detect 15-cm bricks in the road at 150-m range, right?
So, here is the build from the ground up, walking through design choices and constraints, and pointing to source code and hardware I used so you can follow along with the series and learn by doing. For those who prefer watching videos over reading text, I made an 8-minute Loom video that contains most of the stuff in this newsletter.
Step 1: The Architecture
The depth precision scales proportionally with the instantaneous field of view (IFOV, the angle subtended by one pixel) and inversely with the baseline (the distance between the cameras). The math is here. The IFOV is usually fixed by the application for the desired field of view and camera format, so the adjustable parameter is the baseline. To get super-duper high-quality point clouds, you want a gigantic baseline. But why not a 1000-meter baseline? What is the upper limit to baseline? Well, that is what is called “Zmin” or the minimum depth returned by the stereo vision sensor. The larger the disparity search range, the smaller (the better) the Zmin. At NODAR, we search 1024 pixels of disparity to achieve a very small Zmin, whereas other algorithms might only search 64 or 128 pixels to save computational resources. This means that we can see 8 to 16 times closer, so that you only need two instead of three or four cameras to cover the range of interest.
Here is a handy calculator (webapp) to compute the range resolution to help you select the depth precision and Zmin. You can adjust the stereo vision camera resolution, field of view, and the baseline. The “first detection” line in the visualizer is Zmin. Screenshot below:
For this build, I built three stereo vision cameras with 16 mm (30-deg FOV), 25 mm (~19-deg FOV), and 35 mm (~14-deg FOV) focal length lenses. These lenses were selected to be “IR corrected”, that is, the focal point is the same at visible (450-650 nm) and NIR (850 nm) wavelengths. NIR illumination is used at night to avoid blinding humans.
To summarize, the stereo vision architecture design is simple.
Decide on sensor format: 5.4 MP in this case, given the availability of high-resolution HDR (120 dB) imagers with good quantum efficiency (2-3 e- readout noise), an RGB Bayer filter with NIR sensitivity, and decently large pixels for good light collection (3 µm). Smaller f-numbers are generally better for night performance. Don’t worry too much about depth of view for small f-numbers – most autonomy applications have the lens focused at infinity.
Decide on baseline width (distance between the cameras) that balances Zmin with depth precision. Use our web app calculator above if you don’t want to do any math. Of course, the platform needs to be able to physically support the baseline. We’ve deployed up to 5.1-m baselines in practice, for example, in the tail of Regent’s Seaglider. You can read about Regent’s case study here.
Done!
Step 2: The Build
The construction process is quite an enjoyable story, though I will focus on the most important highlights.
First, buy some lenses. IR-corrected lenses are relatively expensive ($500-ish each), but sometimes you get what you pay for.
Kowa IR-corrected lenses with 15, 25, and 35 mm focal lengths. These are long-focal-length “telephoto” lenses.
Next, attach the lenses to the cameras and add bases to the cameras to mount on an extruded aluminum bar. Try to efficiently use your workbench. Lots of little pieces!
The workspace with some of my favorite tools: a Pana-Vise to hold cameras without scratching them, and Wera hex wrenches with a slightly convex head for extra bite.
I assembled the cameras and tried to view an equipment rack at the end of the lab with the lights off and illuminated the rack with some 850-nm LEDs (NIR). Didn’t see a thing. Oops, forgot about the IR blocking filter in the camera housing. The filters had to be removed.
Need an NIR illuminator? Look no further than your $35 security camera for a set of four bright 850-nm LEDs. Economical and effective!
In our cameras, there’s a rectangular glass IR-blocking window, 1-mm thick, that is fastened to the camera housing before the CMOS sensor with some screws and a holding ring. With a precision screwdriver, these came out easily.
Before-and-after images showing the removal of the NIR filters from the cameras. Image of filters, retaining ring, and screws. Manufacturer's warranty officially voided.
Then I put the lenses on the camera and focused them at infinity… or at least I tried to. The focus ring hit a mechanical stop, but the images were still blurry, and I couldn’t focus the image no matter how I changed focus of the lens.
Then I realized that by removing the 1-mm-thick NIR glass, I changed the optical path length in the camera, and moved the infinity-focus plane behind the CMOS sensor! I needed to add another piece of glass back to the path or shorten the path length in air by around 0.5 mm. Getting a custom rectangular glass piece with anti-reflection coatings would take too long and be too expensive for only 6 cameras, so I decided to mechanically remove 0.5 mm from the C-mount housing. The sensitive parts were taped with Kapton tape, and the front of the camera housing was sanded with 80-grit sandpaper (then 200-grit to smooth the edges). I used one 8.5x11-inch sheet of sandpaper for each camera housing, and sanded for about 5 minutes to remove approximately 0.5 mm of the barrel. Using the threads as a depth gauge, as C-mount lenses have 1/32-inch (0.8-mm) thread spacing, I could see that I sanded off about one thread and knew that I was done. To keep the sanded surface level (rather than sanding a ramped surface), I sanded in a figure-8 pattern. Here’s the result.
Sanding is a dirty process. Do it outside. Also, don’t forget to tape up all connectors. You don’t want aluminum shavings in the connectors or on the CMOS sensor.
Now it was time to focus the cameras at infinity. I pointed them to a tree that was far away (about 50 meters) and adjusted the focus until the image was sharp for all six cameras. The iris was set to the maximum opening to maximize light collection in low-light conditions. These are HDR cameras, so I was not worried about over-exposure during the daytime.
Next, I had to measure the cameras’ intrinsic parameters. If you don’t have expensive test equipment, my favorite “checkerboard” implementation is done by Robert Leo, and his GitHub repo, called “Lensboy”, is here. Load your charuco board images into his Jupyter notebook, hit run, and out come intrinsic camera parameters. It shows how well your checkerboard covers the image, and corrects for non-planar checkerboards. Well done! For longer focal lengths, you need bigger checkerboards. You can print out huge 2 x 8-foot self-adhesive posters at Walgreens for ~$50. You can re-stick them on another surface, and they claim that it doesn’t leave a residue on the wall (still haven’t tested that). Get the charuco pattern from calib.io here. Place the self-adhesive poster on a flat surface. I used the concrete wall in my garage, but any drywall is sufficiently flat. Remember, Lensboy corrects for some curvature of the calibration board.
At NODAR, we’re lucky to have an Image Engineering GEOCAL, which is a collimated red laser incident upon a diffractive optic grating. It generates a grid of points, kinda like pointing your cameras at the night sky and using the stars as point sources at infinity, except all the stars are perfectly aligned in a grid. In a single image, you can see the pincushion, barrel, or other distortion. Image Engineering’s software interprets the image and returns the values for the desired camera model. For example, one can specify a pinhole model and get the focal lengths, piercing points, and radial and tangential distortion coefficients. Since these cameras have a long focal length, the pinhole model (“Even Brown Model”) does a good job at capturing the distortion of the lenses. The GEOCAL is a great tool if you need to calibrate lots of cameras or study how intrinsic camera parameters change over time.
Camera intrinsic calibration using the Image Engineering GEOCAL instrument. No worries, we took off the lens caps before taking the measurement.
The next part was to mount all the cameras on a piece of extruded aluminum (sometimes called 80/20). We mounted them at around a 50-70-cm baseline per customer request. The baseline length was limited in this case from the available space to mount the cameras on a motorized pan-tilt mount.
Six cameras. Three stereo pairs. Pure awesomeness.
Finally, we did the extrinsic camera calibration. That was pretty easy. Just measured the distance between the cameras with a tape measure and used the “initial calibration” routine in NODAR Viewer, as explained here.
Step 3: The Test
In the next issue, we’ll show some test results using this rig. For now, enjoy data collected at Boston Harbor using our 1-m baseline, 10.8MP (2 x 5.4MP, dual RGB, and HDR), 65-deg system (Part number NDR-HDK-2.0-100-65-A) and full video here. 3D water reconstruction at its finest!
A boat passing through the field of view, with clearly visible waves and ripples in the water.