Image Segmentation with a challenging background

Question

I'm working on an animal classification problem, with the data extracted from a video feed. The recording was made in a pen, so the problem is quite challenging with a dark background and many shadows:

Initially I tried scikit-image, but then someone helped me with an advanced tool called crf-rnn (http://crfasrnn.torr.vision/) that does a great job segmenting and labelling objects in an image. I did the following:

import caffe
net = caffe.Segmenter(MODEL_FILE, PRETRAINED)
IMAGE_FILE = '0045_crop2.png'
input_image = caffe.io.load_image(IMAGE_FILE)
from PIL import Image as PILImage
image = PILImage.fromarray(np.uint8(input_image))
image = np.array(image)
mean_vec = [np.mean(image[:,:,vals]) for vals in range(image.shape[2])]
im = image[:, :, ::-1]
im = im - reshaped_mean_vec
cur_h, cur_w, cur_c = im.shape
pad_h = 750 - cur_h
pad_w = 750 - cur_w
print(pad_h, pad_w, "999")
im = np.pad(im, pad_width=((0, max(pad_h,0)), (0, max(pad_w,0)), (0, 0)), mode = 'constant', constant_values = 255)
segmentation = net.predict([im])
segmentation2 = segmentation[0:cur_h, 0:cur_w]

The resulting image segmentation is rather poor (although two cows are recognized correctly):

I use a trained crf-rnn (MODEL_FILE, PRETRAINED), which works well for other problems, but this one is harder. I would appreciate any suggestions on how to pre-process this sort of image to extract the shape of most cows.

Answer 1

It would be appreciated if you could explain precisely what your goal is:

1. you want to identify what animal is in your picture ?
2. you want to count the number of animals ?
3. you want to get the position of each animal in the picture ?

In any case, I know that you can get some already trained neural nets from google or anywhere else. This neural net can be used with caffe as it is the case in this google deepdream stuff on github (look at ):

https://github.com/google/deepdream/blob/master/dream.ipynb

Then, if you want to highlight or identify the positions of your animals, you'll find this article inspiring:

http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf

It explain how to reverse convolutional networks to identify what part of your image helped to recognize what is inside. The found projection get you something similar to your second picture (called a mask), but depending on the neural net you use, you can get better results.