I want to use ResNet-18 for image features at the head of a VAE. If I have 10 seconds of video at 30 frames per second, and if ResNet-18 gives me the features of a single frame as a 512-dimensional vector, should I represent my 10-second video features
(300, 512) or
(10, 30, 512).
My intuition is
(300, 512), as I don't want to assume any independence at the 1-second level.