1
$\begingroup$

I want to apply a CNN to a series of image sequences to classify that sequences of frames/images in two groups/categories. We are in a binary classification problem. My dataset is composed by a lot of 'batches' of frames. For example, each batch of frames could be compose of 20 frames of 64x64 pixels. One important thing is that the order of that 20 frames is important. If you shuffle the order of that 20 frames the output could change.

For all this, I want to create a CNN to solve this binary classification problem. I'm usig Keras and TensorFlow.

What is my question? Well, I'm not sure if I have to use a TimeDistributed layer or not. The input shape of the neural network is the following one: (20, 64, 64, 1), whose meaning is: 20 frames with a 64x64 size (1 channel - grayscale). Should I use a TimeDistributed layer?

$\endgroup$
2
  • $\begingroup$ Have you tried TimeDistributed? What alternatives are you considering (a Conv3D layer may also work, as might LSTM)? Typically "what should I try" questions when the choice is not obvious are answered with "try them all and measure how well they do". Would that work for you? $\endgroup$ Oct 1 '17 at 8:41
  • $\begingroup$ I think that what I need to know before is to understand conceptually what is the difference between using or not the TimeDistributed layers. My question is about the conceptual model to resolve the problem. What is the difference between using TimeDistributed layers or not with inputs of several frames instead of frame by frame. In the future, I will try with LSTM and other RNN but as a first stage I need to understand deeply how it works the model without RNN. $\endgroup$ Oct 1 '17 at 8:59
1
$\begingroup$

I'm not sure if I have to use a TimeDistributed layer or not

You definitely don't have to use TimeDistributed. You have other choices, that may be equally valid, depending on your data:

  • Flatten your example data into 81920 features per example and use a simple Dense layer.

  • Use a Conv3D layer.

  • Use some form of RNN, such as LSTM.

From your data description, I would expect either a TimeDistributed or 3D CNN based approach to be a good first bet. Intuition suggests that the CNN would work better if there was minor change between frames (because it has capability to directly find subtle frame differences), the TimeDistributed approach would work better processing larger changes (because it will ignore frame differences until the fully-connected layer).

Should I use a TimeDistributed layer?

Only you can answer this, by trying it and measuring performance of your classifier. However, it should function correctly, and intuition suggests it would be a good choice if your frames are in sequence but visually disjoint.

$\endgroup$
2
  • $\begingroup$ If the CNN+TimeDistributed approach would work better processing larger changes and the CNN without the TimeDistributed layer would work better if there was minor change between frames, I think that the second is where I have to start. I didn't know that TimeDistributed works better processing larger changes. Thank you so much for your advice. $\endgroup$ Oct 1 '17 at 13:03
  • $\begingroup$ It's probably more accurate to state that TimeDistributed processes frames separately but identically, whilst Conv3D processes changes between frames similar to how it processes changes between pixels within a frame. This leads to them having different strengths/weaknesses. My intuition is that a slideshow sequence of different subjects (or with large time intervals) is likely to play well to TimeDistributed's strengths, whilst a short video clip is likely to work better with Conv3D. If you have time, test both, I don't know your data! $\endgroup$ Oct 1 '17 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.