I want to use ResNet-18 for image features at the head of a VAE. If I have 10 seconds of video at 30 frames per second, and if ResNet-18 gives me the features of a single frame as a 512-dimensional vector, should I represent my 10-second video features (300, 512)
or (10, 30, 512)
.
My intuition is (300, 512)
, as I don't want to assume any independence at the 1-second level.