![\"photo](\"https:\/\/ece.ncsu.edu\/wp-content\/uploads\/2022\/01\/tianfu-wu-monocon-header-1024x576.jpg\")
<\/p>\n<\/p>\n
Researchers have developed a new technique, called MonoCon, that improves the ability of artificial intelligence (AI) programs to identify three-dimensional (3D) objects, and how those objects relate to each other in space, using two-dimensional (2D) images. For example, the work would help the AI used in autonomous vehicles navigate in relation to other vehicles using the 2D images it receives from an onboard camera.<\/p>\n
\u201cWe live in a 3D world, but when you take a picture, it records that world in a 2D image,\u201d says Tianfu Wu, corresponding author of a paper on the work and an assistant professor of electrical and computer engineering at North Carolina State University.<\/p>\n
\u201cAI programs receive visual input from cameras. So if we want AI to interact with the world, we need to ensure that it is able to interpret what 2D images can tell it about 3D space. In this research, we are focused on one part of that challenge: how we can get AI to accurately recognize 3D objects \u2013 such as people or cars \u2013 in 2D images, and place those objects in space.\u201d<\/p>\n
While the work may be important for autonomous vehicles, it also has applications for manufacturing and robotics.<\/p>\n
In the context of autonomous vehicles, most existing systems rely on lidar \u2013 which uses lasers to measure distance \u2013 to navigate 3D space. However, lidar technology is expensive. And because lidar is expensive, autonomous systems don\u2019t include much redundancy. For example, it would be too expensive to put dozens of lidar sensors on a mass-produced driverless car.<\/p>\n
\u201cBut if an autonomous vehicle could use visual inputs to navigate through space, you could build in redundancy,\u201d Wu says. \u201cBecause cameras are significantly less expensive than lidar, it would be economically feasible to include additional cameras \u2013 building redundancy into the system and making it both safer and more robust.<\/p>\n
\u201cThat\u2019s one practical application. However, we\u2019re also excited about the fundamental advance of this work: that it is possible to get 3D data from 2D objects.\u201d<\/p>\n
Specifically, MonoCon is capable of identifying 3D objects in 2D images and placing them in a \u201cbounding box,\u201d which effectively tells the AI the outermost edges of the relevant object.<\/p>\n
MonoCon builds on a substantial amount of existing work aimed at helping AI programs extract 3D data from 2D images. Many of these efforts train the AI by \u201cshowing\u201d it 2D images and placing 3D bounding boxes around objects in the image. These boxes are cuboids, which have eight points \u2013 think of the corners on a shoebox. During training, the AI is given 3D coordinates for each of the box\u2019s eight corners, so that the AI \u201cunderstands\u201d the height, width and length of the \u201cbounding box,\u201d as well as the distance between each of those corners and the camera. The training technique uses this to teach the AI how to estimate the dimensions of each bounding box and instructs the AI to predict the distance between the camera and the car. After each prediction, the trainers \u201ccorrect\u201d the AI, giving it the correct answers. Over time, this allows the AI to get better and better at identifying objects, placing them in a bounding box, and estimating the dimensions of the objects.<\/p>\n
\u201cWhat sets our work apart is how we train the AI, which builds on previous training techniques,\u201d Wu says. \u201cLike the previous efforts, we place objects in 3D bounding boxes while training the AI. However, in addition to asking the AI to predict the camera-to-object distance and the dimensions of the bounding boxes, we also ask the AI to predict the locations of each of the box\u2019s eight points and its distance from the center of the bounding box in two dimensions. We call this \u2018auxiliary context,\u2019 and we found that it helps the AI more accurately identify and predict 3D objects based on 2D images.<\/p>\n
\u201cThe proposed method is motivated by a well-known theorem in measure theory, the Cram\u00e9r\u2013Wold theorem. It is also potentially applicable to other structured-output prediction tasks in computer vision.\u201d<\/p>\n
The researchers tested MonoCon using a widely used benchmark data set called KITTI.<\/p>\n
\u201cAt the time we submitted this paper, MonoCon performed better than any of the dozens of other AI programs aimed at extracting 3D data on automobiles from 2D images,\u201d Wu says. MonoCon performed well at identifying pedestrians and bicycles, but was not the best AI program at those identification tasks.<\/p>\n
\u201cMoving forward, we are scaling this up and working with larger datasets to evaluate and fine-tune MonoCon for use in autonomous driving,\u201d Wu says. \u201cWe also want to explore applications in manufacturing, to see if we can improve the performance of tasks such as the use of robotic arms.\u201d<\/p>\n