I am working with a multimodal dataset, it's a video dataset with a total video 10400. For the audio data I have it turned into a log melspectrogram and the spectrogram have the following shape (timesteps, num_features) = (190, 96) and then I used the resnet18 architecture and it worked. Now for the visual data, it has the following shape (num_frames, height, width) = (75, 100, 50) however when I tried using resnet18 on it the validation loss was stuck while the training loss keeps going down, so I guess it must have overfit? The same thing happened even when I tried with initial number of filters of 1. And I have tried doubling the amount of data with horizontal flip yet the same thing still happens. Does resnet18 really is too powerful for the task? But if it does why does it work with the audio and not the visual?