# Image Segmentation with a challenging background

I'm working on an animal classification problem, with the data extracted from a video feed. The recording was made in a pen, so the problem is quite challenging with a dark background and many shadows:

Initially I tried scikit-image, but then someone helped me with an advanced tool called crf-rnn (http://crfasrnn.torr.vision/) that does a great job segmenting and labelling objects in an image. I did the following:

import caffe
net = caffe.Segmenter(MODEL_FILE, PRETRAINED)
IMAGE_FILE = '0045_crop2.png'
from PIL import Image as PILImage
image = PILImage.fromarray(np.uint8(input_image))
image = np.array(image)
mean_vec = [np.mean(image[:,:,vals]) for vals in range(image.shape[2])]
im = image[:, :, ::-1]
im = im - reshaped_mean_vec
cur_h, cur_w, cur_c = im.shape
segmentation = net.predict([im])
segmentation2 = segmentation[0:cur_h, 0:cur_w]


The resulting image segmentation is rather poor (although two cows are recognized correctly):

I use a trained crf-rnn (MODEL_FILE, PRETRAINED), which works well for other problems, but this one is harder. I would appreciate any suggestions on how to pre-process this sort of image to extract the shape of most cows.

It would be appreciated if you could explain precisely what your goal is:

1. you want to identify what animal is in your picture ?
2. you want to count the number of animals ?
3. you want to get the position of each animal in the picture ?

In any case, I know that you can get some already trained neural nets from google or anywhere else. This neural net can be used with caffe as it is the case in this google deepdream stuff on github (look at ):

Then, if you want to highlight or identify the positions of your animals, you'll find this article inspiring:

http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf

It explain how to reverse convolutional networks to identify what part of your image helped to recognize what is inside. The found projection get you something similar to your second picture (called a mask), but depending on the neural net you use, you can get better results.

• 'what your goal is': point 2+3; he problem is that the background in hard to extract, as there are a lot of shadows and many cows are of a similar color
– Alex
Mar 22 '16 at 10:42
• Well, your image still have some good edges. You can try using some edge detection algorithm to accentuate the objects limits: en.wikipedia.org/wiki/Edge_detection. But once again, I think the current network your working with is just not good enough for this case. Did you try some basic preprocessing like contrast or spectral regularization ? Mar 22 '16 at 11:56
• Do you mean like this: scikit-image.org/docs/dev/api/…
– Alex
Mar 22 '16 at 14:44
• I think edge-based algorithms are more adapted since neural networks and image recognition are mostly based on edges (and not colors). So you should try amplifying the edges on your image. Mar 22 '16 at 20:39
• I've had a bit of success with histogram equalization to increase contrast, but otherwise not sure how to continue. What can I do to maximize the effect of a segmentation algorithm?
– Alex
Mar 22 '16 at 21:06