Researchers leverage inkjet printing to make a portable multispectral 3D camera

3D spectral information can be useful for classifying objects and materials; however, capturing 3D spatial and spectral information from a scene typically requires multiple devices or time-intensive scanning processes. The new light field camera solves this challenge by simultaneously acquiring 3D information and spectral data in a single snapshot.

"To our knowledge, this is the most advanced and integrated version of a multispectral light field camera," said research team leader Uli Lemmer from the Karlsruhe Institute of Technology in Germany. "We combined it with new AI methods for reconstructing the depth and spectral properties of the scene to create an advanced sensor system for acquiring 3D information."

In the Optica Publishing Group journal Optics Express, the researchers report that the new camera and image reconstruction methods can be used to distinguish objects in a scene based on their spectral characteristics. Using inkjet printing to make the camera's key optical components allows it to be easily customized or manufactured in large volumes.

"Reconstructed 3D data from camera images are finding widespread use in virtual and augmented reality, autonomous cars, robotics, smart home devices, remote sensing and other applications," said Michael Heizmann, a member of the research team. "This new technology could, for example, allow robots to better interact with humans or improve the accuracy of classifying and separating materials in recycling. It could also be potentially used to classify healthy and diseased tissues."

Adding color with inkjet printing

Light field cameras, also called plenoptic cameras, are specialized imaging devices that capture the direction and intensity of light rays. After image acquisition, computational processing is used to reconstruct 3D image information from the acquired data. These cameras typically use microlens arrays that are aligned with the pixels of a high-resolution camera chip.

To create a multispectral light field camera, the researchers used inkjet printing to deposit a single droplet of material to form each individual lens on one side of ultrathin microscope slides and then printed fully aligned color filter arrays on the opposite side of the microscope slides. The resulting optical component was integrated directly onto a CMOS camera chip. The inkjet printing method allowed precise alignment between the optical components, significantly reducing the manufacturing complexity and enhancing efficiency.

Because this setup produces spectral and depth information that are interwoven in the camera image, the researchers developed methods to separate each component. They found that an approach based on deep learning worked best for extracting the desired information directly from the acquired measurements.

Spectral-based object detection

"Tackling the challenge of creating a multispectral light field camera was only possible by combining recent advances from manufacturing, system design and AI-based image reconstruction," said Qiaoshuang Zhang, first author of the paper. "This work pushes the boundaries of inkjet printing -- a versatile method with high precision and industrial scalability -- for manufacturing photonic components."

The researchers tested the camera by recording a test scene that contained multicolor 3D objects at different distances. The image reconstruction algorithm was trained and tested on many synthetic and real multispectral images. The results demonstrate that the prototype camera can simultaneously acquire 3D spatial and spectral information and that different objects can be imaged and distinguished by their different spectral composition and depth information within one single snapshot.

Now that they have completed this first proof-of-concept, the researchers are exploring various applications where a light field camera with an ability to acquire multispectral information could be useful.