Video classification of birds

Question

I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked for my dataset.

Dataset

The dataset consists of 9 different classes with around 3 000 videos with labels:

• Björktrast (Fieldfare)
• Koltrast (Blackbird)
• Nötväcka (Nuthatch)
• Pilfink (Tree sparrow)
• Mus (Mouse)
• Blåmes (Blue tit)
• Rödhake (Robin)
• Katt (Cat)
• Inget objekt (No object)

---

Here is a video of a sample dataset containing the different classes that my deep learning model should be able to predict:

Method

Currently I have used this codebase for temporal segment networks to train my model and the resulting accuracy for my own dataset was about 87% (only with RGB and not using optical flow). Although depending on weather condition and lighting, the accuracy can be very different.

I have expanded the dataset to around 30 000 videos using the following techniques:

• Adding RGB noise to videos already in the training dataset.
• Adding random distortions and warping to videos in the training dataset.

Problems

I have a few questions about doing video classification to predict bird species:

• Are there better techniques for doing video classification compared to using temporal segment networks?
• Are there any preprocessing steps that should be done to improve accuracy?
• How to train the model to generalize better to be less sensitive to lighting and weather?
• How can the model be trained to classify unknown species?
• Are there other ways to expand the dataset except those mentioned above?

In short: What is the best deep learning model for video classification for bird species?

Update 2018/04/17: I discovered that the accuracy of the model is significantly lower with objects obstructing the camera which have similar colours to the classes of birds to be recognized. This often causes the model to label the video incorrectly and setting a high confidence.

Answer 1

Are there better techniques for doing video classification compared to using temporal segment networks?

I would suggest to train the network to detect bird species in image space without the temporal dimension. This will make the network have much less parameters to tune and you will likely get better results. Then for a novel video, you can detect the species for each frame and do a majority vote to decide what species was in the video.

Are there any preprocessing steps that should be done to improve accuracy?

If you train in image space then you can use a DataGenerator on the image and transform it in all kinds of ways, the position and rotation of the bird does not affect the species. This can be done through the Keras pipeline. You can check how to do this in general here.

How to train the model to generalize better to be less sensitive to lighting and weather?

You can try to pass your images through some pretrained segmentation network to identify background and foreground and then set that area to a static color. Perhaps black (i.e. all 0's). This will make the decision boundary for the network easier. However, I think you should focus on the other points, the network should inherently detect what features are the most salient for discriminating different species, if you have sufficient training samples the weather and background will be low information features and will thus not be used.

How can the model be trained to classify unknown species?

This would be called anomaly detection. This is a much harder problem. If you have birds of other species available in your dataset then you can add a class label for other. This is still standard supervised learning. If this is not available it will be very difficult. Most anomaly detection algorithms do not work well for high dimensional datasets. Then you will need a way to compress the information in your input images. This can be achieved with some transfer learning on a pretrained autoencoder network trained on natural images such as ImageNET, then you can use transfer learning using your bird images to further tune the network for your specific use case. You can then use this compressed feature space using one of the anomaly detection techniques I described here.