Glass bottles and crystal bowls bend light in strange ways. Image processing networks often struggle to separate the boundaries of transparent objects from the background that shows through them. A new method sees such items more accurately.

What’s new: Shreeyak Sajjan and researchers at Synthesis.ai, Google, and Columbia University premiered a state-of-the-art model for identifying transparent objects. They call it ClearGrasp, a reference to its intended use in robotics.

Key insight: Faced with a transparent object, RGB-D cameras, which sense color and depth per pixel, can get confused: They take some depth measurements off the object’s surface, others straight through the object. ClearGrasp recognizes such noisy measurements and uses them to predict an object’s shape. Once it knows the object’s shape and how far away one point is, it can infer how far away every point is.

How it works: ClearGrasp incorporates a trio of Deeplabv3+ models with the DRN-D-54 architecture.

ClearGrasp’s training dataset included 18,000 simulated and 22,000 real images. To make the real images, the researchers photographed transparent objects, yielding depth measurements that encoded distorted light passing through them. Then they painted the objects and photographed them again to obtain accurate depth measurements.
The first Deeplabv3+ model removes depth measurements associated with transparent objects, retaining data on opaque objects, which presumably is accurate. The second extracts approximate object boundaries. The third generates improved depth measurements.
ClearGrasp combines the three outputs to get accurate depth measurements of both foreground and background.

Results: Fed real-life data captured by the researchers, ClearGrasp improved the previous state of the art’s root mean squared error of corrected depth measurements from 0.054 to 0.038. A robotic arm using ClearGrasp picked up transparent objects 72 percent of the time, a big step up from 12 percent using unprocessed images.

Why it matters: Machine learning has proven to be adept at noise reduction in various domains. ClearGrasp takes special care to modify only the depth measurements that are distorted.

We’re thinking: ClearGrasp could prevent your robot assistant from having to clean up broken glass all day.

ClearGrasp

Seeing the See-Through

Subscribe to The Batch