2
$\begingroup$

I know it might be a generic question but I would still appreciate some feedback. So I have a dataset with 4 dimensions (time, x, y, color). Where I have a total of 24000 records each with (5, 188, 188, 3) this dimension where I have a 5-time dimension which is like 5 different time intervals each at 15 minutes difference. Now I have to build a model for binary classification from the image. So initially I used Resnet with 3 dimensions from data (x, y, color), which gave a pretty decent result. Now I also want to include the time dimension in my data but I'm not sure RESNET will make any sense if the time dimension is included. Can someone tell me if there's another network that I can try where the model can make use of time dimension, or adding a layer like LSTM in the model can be helpful? I have attached a sample gif with 5-time dimensions and 1 color. Any suggestion will be appreciated.

Sample data:

[[[0.1798448 , 0.1735874 , 0.1735874 , ..., 0.04843931,
         0.05156801, 0.05469671],
        [0.17671609, 0.1798448 , 0.1829735 , ..., 0.05156801,
         0.05313236, 0.05156801],
        [0.18140915, 0.1861022 , 0.18140915, ..., 0.05156801,
         0.05313236, 0.05156801],
        ...,
       [[0.17515175, 0.17045869, 0.1735874 , ..., 0.05626106,
         0.05000366, 0.05626106],
        [0.17515175, 0.1735874 , 0.17202304, ..., 0.06564717,
         0.06408282, 0.06721152],
        [0.1735874 , 0.1735874 , 0.16889434, ..., 0.06721152,
         0.06251846, 0.06095411],
        ...,
        [0.06877588, 0.07034022, 0.06877588, ..., 0.01715229,
         0.01558794, 0.01715229],
        [0.07346892, 0.07190457, 0.06721152, ..., 0.01871664,
         0.01715229, 0.02028099],
        [0.07503328, 0.06721152, 0.06564717, ..., 0.01558794,
         0.02184534, 0.02184534]],

       [[0.1735874 , 0.17045869, 0.16889434, ..., 0.04843931,
         0.05313236, 0.05938977],
        [0.17515175, 0.17202304, 0.16732998, ..., 0.05938977,
         0.05938977, 0.06408282],
        [0.17045869, 0.16889434, 0.16576564, ..., 0.06564717,
         0.06251846, 0.06564717]]], dtype=float32)

enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ You have a color video, right? $\endgroup$
    – Dave
    Commented Jun 17, 2021 at 1:42
  • $\begingroup$ @Dave not exactly color video, I have 4d array as mentioned above where 1st dimension is time and last is color channel. I created gif file from that using 5 time dimension and 1 color channel for vizulization. $\endgroup$
    – Chris_007
    Commented Jun 17, 2021 at 2:00
  • $\begingroup$ I do not follow your last sentence about five times dimensions and one color channel, but if your data are a bunch of gifs, I would call that video-like. $\endgroup$
    – Dave
    Commented Jun 17, 2021 at 2:59
  • $\begingroup$ @Dave yup it’s like that. Can you suggest me something that I should try? Any suggestions will be appreciated! :) $\endgroup$
    – Chris_007
    Commented Jun 18, 2021 at 11:23

1 Answer 1

Reset to default
2
$\begingroup$

In your case, it seems that time has a clear meaning because you have a movement logic between images. As a consequence, I would recommend models adapted to videos like MetNet, ConvLSTM or PredRNN... https://ai.googleblog.com/2020/03/a-neural-weather-model-for-eight-hour.html

https://proceedings.neurips.cc/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html

https://proceedings.neurips.cc/paper/2017/file/e5f6ad6ce374177eef023bf5d0c018b6-Paper.pdf

Hope this helps,

Nicolas

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you so much! This is what exactly I was looking for. By any chance can you share any implementation links if you have any! $\endgroup$
    – Chris_007
    Commented Jun 17, 2021 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.