I am making a custom image classifier using Transfer Learning on Inception V3. I have 3 classes of images with ~6K images each. The input dimension of the network is 500X500 and the output of the network is 14X14x2048. I used global average pooling and finally got a vector of size 2048.
To get just a baseline model, I initially trained a linear SVM classifier on this feature vector, (PS: The network has not been trained for this dataset, right now I am just using the forward pass on the Imagenet weights).
The Linear SVM Classifier was giving an accuracy of 81%. (Which was a little surprising, but I guess the pre trained weights were able to capture a lot of distinctions in those images).
Finally, to at least replicate the results using neural networks, I made all the layers up till the 2048 feature vector untrainable, and straight away mapped it to a 3 output softmax activated output layer (which was the only trainable layer).
After putting the training on for ~400 epochs, the validation accuracy remains fluctuating between 33%-38% which is absolutely surprising. It goes on to implicate that SVM classifier on the same feature vectors is able to learn the features which neural networks aren't able to.
Is there any suitable explanation for this phenomenon?