# Combining 2D Detection with Disparity Maps to Learn 3D Object Geometry

Since the disparity map above is a representation of the object's distance from the camera's origin, is it reasonable to assume that a network (perhaps a convolutional LSTM) could be trained to perform a infer the depth of the object in the Z coordinate of camera frame?

As you can see above the brightness of the object increases as the car moves closer to it, and the size of the 2D bounding box also increases.

Thus the depth in the Z direction becomes a function of two things:

1. the change in the diagonal/size of the 2D bounding box

1.1. More or less the object's size as the car approaches it because it can also be moving across from that car as it is stopped at a street light. In which case the 2D box is mostly constant in size.

2. the change of the brightness of the pixels inside the 2D box.

Before going down this rabbit hole, I was hoping to get feedback on the feasibility of this idea, what "gotcha's" there might be that are not immediately apparent.

A couple of things are immediately noticeable about why this would be too difficult:

1. The occlusion of parked cars on the right hand side there makes their shapes very hard to distinguish, although maybe the 2D boxes could help with that
2. It will be difficult to track the same object through time steps, and would probably require a SLAM network as well.

Any help is very much appreciated!

• Just thinking about this, it might be very critical to base the distance function conditioned on the bounding box. Imagine you miss one Object, so you dont have a bounding box, you automatically will lose all depth information – Jens K Nov 12 '19 at 14:04
• @JensK Agreed. I am thinking the input will be a column vector where the first two elements are the height and width of the bounding box, followed by X number of pixels (from disp map). The problem is X is the product of HxW of the box, so I need to figure out how to had variable length input to a network. About to ask that question here and will reply with link – Sam Hammamy Nov 12 '19 at 14:13
• If I understand correctly you have a vector indicating the size of the window followed by all pixels in that window, so [h,w, x_1, x_n]. You could reorder x_1,...x_n to be just the cropped box and then apply a fully convolutional network which can operate on arbitrary sizes, if the striding matches. So you may want to resize the boxes to the next fitting size. Then you can predict every Box at every time frame to get the depth information. And yes you are right, you will have to track the same object in ervery frame to make this work. – Jens K Nov 12 '19 at 14:40
• I think you worded better than I did! Yes it would be x_1...x_n where each element is the value of that pixel. I think your idea of fully convnets would work best. It is a little weird though because the disp image is uniform with very little edge information. But this is a really good place to start! – Sam Hammamy Nov 12 '19 at 14:50