2
$\begingroup$

I have certain videos for which the frames are labeled either as dirty (meaning the camera lens is occluded by soil or rain) or as clean. The goal is to test a convolutional neural net on this data to evaluate how well it can classify if a frame is dirty or clean.

One idea is to use the first layers of an existing network and train last (fully connected) layers using the available data.

But one issue might be that basically all available networks are trained for object classification and might not be very suitable for tasks like soil and rain detection.

Do you have any recommendations about networks or models which might be suitable for this task?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ It is possible that someone here has done something similar enough that they can advise you of good pre-trained models or special caveats. However, you may have to accept that you are looking at a niche problem where your own experimental results are the only thing you have to guide you. The usual approach is to try stuff and measure the accuracy (or other metric). Starting with a pre-trained network is sensible if you do not have lots of your own training examples. But you would also be wise to try your own CNN and train from scratch for comparison. Find the best solution scientifically $\endgroup$ – Neil Slater Dec 12 '16 at 15:34
  • $\begingroup$ How many tagged images (frames) do you have? $\endgroup$ – Armen Aghajanyan Dec 13 '16 at 6:42
  • $\begingroup$ I have a few hours of video with 30 frames per second $\endgroup$ – johnny b Dec 13 '16 at 12:54
  • $\begingroup$ @johnnyb By the way, I have a very similar dataset, which are short videos recorded on my dash cam while I drove around town. There are sunny days and rainy days. But the issue I face is how to deal with, or get ride of the frames where the windshield wiper crosses the camera. $\endgroup$ – horaceT Jan 13 '17 at 19:54
1
$\begingroup$

So neural nets can be quite powerful for end-to-end training. So if you have a dataset that contains both clean and dirty images, and also the classes you are trying to detect in the first place, it should be fairly straightforward to try to train you net.

I'd proceed as follows.

  1. adjust the network to produce a logits vector that outputs not only the object classification but also a binary classifier for whether your image is clean or dirty.
  2. then adjust the labels for each image to indicate the classification label and the cleanliness of your image.
  3. I'd then train and test on the dataset to see if a model emerges.

The convolutional parts of your network should converge to features that are good at
A. classifying the subject of the image, and B. detecting whether the camera is dirty or not.

Its a good idea and I might use it myself!

Improve this answer
$\endgroup$
0
$\begingroup$

This is a great problem. Rain affects the image through a localized lensing effect, so I'd partition the image into patches big enough to capture a raindrop, then run a classifier on each patch. Training data is not an issue since you say you have video. I'm not sure how to capture this lensing effect, but my intuition would be to use a 2D FFT or wavelet decomposition to extract the raw features your CNN will run on top off. If this does not work well enough, I suggest asking about what kind of features to use to detect lensing on physics.SE.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I'm not sure that applying a CNN to the output of a 2D FFT will work well. A CNN makes sense when there are spatially local patterns that are meaningful no matter where in the image they appear -- i.e., when there's some translation-invariance. That assumption doesn't hold if you've first run a 2D FFT on the image. I suspect you should either use a CNN on the raw image; or if you're going to do a 2D FFT first, then you'll probably need a different neural network architecture (fully connected?). The first 3 sentences of the answer seem helpful either way. $\endgroup$ – D.W. Dec 15 '16 at 9:05
  • 1
    $\begingroup$ I disagree a little with "Training data is not an issue since you say you have video". Frames from the same scene are going to be highly correlated, so a few hours of video may not actually amount to enough variation to train a general model. If there are thousands of scenes in positive/negative category, each only a few seconds of video, then that is a different case. $\endgroup$ – Neil Slater Dec 15 '16 at 13:18
0
$\begingroup$

There is this prevailing perception that convolution neural net is the panacea to solve all image recognition problems, which understandably comes from the many success stories in the computer vision literature. However, keep in mind what a ConvNet does for you. It finds the translational invariant features, like edges.

From what you described here, a ConvNet is unlikely to do much for you. A raindrop or a dirt spot is equally likely to fall on any location of the video frames. The problem you have is one of texture classification /discrimination, and there is a huge literature on that subject. A google scholar search finds this[1].

[1] Smith, J. R., & Chang, S. F. (1994, November). Transform features for texture classification and discrimination in large image databases. In Image Processing, 1994. Proceedings. ICIP-94., IEEE International Conference (Vol. 3, pp. 407-411). IEEE.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "It finds the translational invariant features, like edges." I do not agree on that. What kind of features it will "find" depends on the input data. E.g. sharpness features are very helpful to discriminate if the lens is dirty or not. This can be learned by a ConvNet. E.g. the convolutional kernels can be similar to Laplacians or Gaussians. $\endgroup$ – johnny b Jan 16 '17 at 13:24
  • $\begingroup$ "There is this prevailing perception that convolution neural net is the panacea to solve all image recognition problems" I agree on that. The point here is just to test ConvNets. If the outcome is that ConvNets do not work well or worse then other algorithms then it is also an useful outcome. $\endgroup$ – johnny b Jan 16 '17 at 13:25
  • $\begingroup$ "The problem you have is one of texture classification /discrimination" It is a good idea though to use texture features. $\endgroup$ – johnny b Jan 16 '17 at 13:25
  • $\begingroup$ "A raindrop or a dirt spot is equally likely to fall on any location of the video frames." That is perfectly fine for a ConvNet as a single convolutional kernel is applied on the whole input with the same weights. $\endgroup$ – johnny b Jan 16 '17 at 13:26
  • $\begingroup$ @johnnyb Points taken. Pls post if you make convnet work on this data. I'm in fact wrestling with many similar issues like yours. $\endgroup$ – horaceT Jan 16 '17 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.