I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked for my dataset.

The dataset consists of 9 different classes with around 3000 videos with labels. I have expanded the dataset to around 30000 videos using the following techniques:

• adding rgb noise to video
• random distortions

Here is a video of a sample dataset containing the different classes that my deep learning model should be able to predict:

![https://www.youtube.com/watch?v=4iNpw7J5q9I]]1

Currently I have used this codebase for temporal segment networks to train my model and the resulting accuracy for my own dataset was about 87% (only with RGB and not using optical flow). Although depending on weather condition and lighting, the accuracy can be very different.

I have a few questions about doing video classification to predict bird species:

• Are there better techniques for doing video classification compared to using temporal segment networks?
• Are there any preprocessing steps that should be done to improve accuracy?
• How to train the model to generalize better to be less sensitive to lighting and weather?
• How can the model be trained to classify unknown species?

In short: What is the best deep learning model for video classification for bird species?

I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked for my dataset.

Dataset

The dataset consists of 9 different classes with around 3 000 videos with labels:

• Björktrast (Fieldfare)
• Koltrast (Blackbird)
• Nötväcka (Nuthatch)
• Pilfink (Tree sparrow)
• Mus (Mouse)
• Blåmes (Blue tit)
• Rödhake (Robin)
• Katt (Cat)
• Inget objekt (No object)

Here is a video of a sample dataset containing the different classes that my deep learning model should be able to predict:

![https://www.youtube.com/watch?v=4iNpw7J5q9I]]1

Method

Currently I have used this codebase for temporal segment networks to train my model and the resulting accuracy for my own dataset was about 87% (only with RGB and not using optical flow). Although depending on weather condition and lighting, the accuracy can be very different.

I have expanded the dataset to around 30 000 videos using the following techniques:

• Adding RGB noise to videos already in the training dataset.
• Adding random distortions and warping to videos in the training dataset.

Problems

I have a few questions about doing video classification to predict bird species:

• Are there better techniques for doing video classification compared to using temporal segment networks?
• Are there any preprocessing steps that should be done to improve accuracy?
• How to train the model to generalize better to be less sensitive to lighting and weather?
• How can the model be trained to classify unknown species?
• Are there other ways to expand the dataset except those mentioned above?

In short: What is the best deep learning model for video classification for bird species?

Update 2018/04/17: I discovered that the accuracy of the model is significantly lower with objects obstructing the camera which have similar colours to the classes of birds to be recognized. This often causes the model to label the video incorrectly and setting a high confidence.