3
$\begingroup$

I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked for my dataset.

Dataset

The dataset consists of 9 different classes with around 3 000 videos with labels:

  • Björktrast (Fieldfare)
  • Koltrast (Blackbird)
  • Nötväcka (Nuthatch)
  • Pilfink (Tree sparrow)
  • Mus (Mouse)
  • Blåmes (Blue tit)
  • Rödhake (Robin)
  • Katt (Cat)
  • Inget objekt (No object)

Here is a video of a sample dataset containing the different classes that my deep learning model should be able to predict:

https://www.youtube.com/watch?v=4iNpw7J5q9I]

Method

Currently I have used this codebase for temporal segment networks to train my model and the resulting accuracy for my own dataset was about 87% (only with RGB and not using optical flow). Although depending on weather condition and lighting, the accuracy can be very different.

I have expanded the dataset to around 30 000 videos using the following techniques:

  • Adding RGB noise to videos already in the training dataset.
  • Adding random distortions and warping to videos in the training dataset.

Problems

I have a few questions about doing video classification to predict bird species:

  • Are there better techniques for doing video classification compared to using temporal segment networks?
  • Are there any preprocessing steps that should be done to improve accuracy?
  • How to train the model to generalize better to be less sensitive to lighting and weather?
  • How can the model be trained to classify unknown species?
  • Are there other ways to expand the dataset except those mentioned above?

In short: What is the best deep learning model for video classification for bird species?

Update 2018/04/17: I discovered that the accuracy of the model is significantly lower with objects obstructing the camera which have similar colours to the classes of birds to be recognized. This often causes the model to label the video incorrectly and setting a high confidence.

$\endgroup$
  • $\begingroup$ Have you read the literature on animal recognition too? Action recognition is rather different. $\endgroup$ – Emre Apr 17 '18 at 2:14
  • $\begingroup$ @Emre any recommendations related to my question? I have looked at this paper researchgate.net/publication/… which looks promising. However, it was used to classify images and hence did not take into account temporal features in a video. $\endgroup$ – Linus Apr 17 '18 at 7:07
  • $\begingroup$ Do you not get better results by just training the database on images. Then for a new video, feed each frame through the network and get a maximum vote for the species in that test video? $\endgroup$ – JahKnows Apr 17 '18 at 8:49
1
$\begingroup$

Are there better techniques for doing video classification compared to using temporal segment networks?

I would suggest to train the network to detect bird species in image space without the temporal dimension. This will make the network have much less parameters to tune and you will likely get better results. Then for a novel video, you can detect the species for each frame and do a majority vote to decide what species was in the video.

Are there any preprocessing steps that should be done to improve accuracy?

If you train in image space then you can use a DataGenerator on the image and transform it in all kinds of ways, the position and rotation of the bird does not affect the species. This can be done through the Keras pipeline. You can check how to do this in general here.

How to train the model to generalize better to be less sensitive to lighting and weather?

You can try to pass your images through some pretrained segmentation network to identify background and foreground and then set that area to a static color. Perhaps black (i.e. all 0's). This will make the decision boundary for the network easier. However, I think you should focus on the other points, the network should inherently detect what features are the most salient for discriminating different species, if you have sufficient training samples the weather and background will be low information features and will thus not be used.

How can the model be trained to classify unknown species?

This would be called anomaly detection. This is a much harder problem. If you have birds of other species available in your dataset then you can add a class label for other. This is still standard supervised learning. If this is not available it will be very difficult. Most anomaly detection algorithms do not work well for high dimensional datasets. Then you will need a way to compress the information in your input images. This can be achieved with some transfer learning on a pretrained autoencoder network trained on natural images such as ImageNET, then you can use transfer learning using your bird images to further tune the network for your specific use case. You can then use this compressed feature space using one of the anomaly detection techniques I described here.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, I will try to implement the suggestions in your answer with github.com/facebook/fb.resnet.torch. Would a model with RNN + CNN give better results so the network doesn't give very different results for each frame? $\endgroup$ – Linus Apr 17 '18 at 10:23
  • $\begingroup$ Also, if I run each frame through the network and then select the species with the most votes, wouldn't this fail if the videos contains less than 50% images of birds and 50% with no object? $\endgroup$ – Linus Apr 17 '18 at 16:14
  • $\begingroup$ Yes it would. You would have to select the frames. You can also have 2 separate networks. One which is generally trained to detect if a frame contains a bird which can be trained with images from ImageNet and then you can have your specialized species detection algorithm. $\endgroup$ – JahKnows Apr 17 '18 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.