I am testing transfer learning on rather small neural networks with only two hidden layers of 20 neurons on tabular data.
None of my experiments yields any improvement over a basic neural network. Is this expected? Does transfer learning only make sense for very large neural networks consisting of many hidden layers?
Are you aware of some studies that use transfer learning successfully for small networks?