I am still fairly new to modern neural networks (dabbled in the 90's when convolution was called neocognition which I didn't explore regrettably).

I read the suggestions after typing in the above title. I don't think my question has been asked quite this way. I'm in (enjoying) hyperparameter search mode. Is there intuition on the rate of converging towards an optima (hopefully global) in gradient descent with respect to guiding me towards a better architecture or away from a poor one?

Details are below. The observation is that between two architecture an unsatisfactory design converges toward optima faster than the second more complex architecture of yet unknown viability. I don't mean faster in time because of course a more complex architecture will take longer to compute. I mean faster/slower rate of change of loss by number of epochs. With the new architecture continuing to reduce in error in the earlier stages of increasing amount of data but seeming to struggle to do so by comparison with the unsatisfactory design. My question more specifically: is this an intuitive indication that the new more complex architecture is a bigger waste of time and should be ditched than the smaller one or is it still worth exploring even though taking orders of magnitude more epochs to drop in loss?

Thanks for getting this far, here's even more detail if you care to read: In my current case where training from scratch I reached a viable seeming primarily convolutional architecture. For initial testing of the waters I trained it on a subset of the full training set and it trained and performed well as additional training data was added. Up to a point. Note that new data was added to the training set without re-initializing the weights, rather considering the network partially pre-trained. With a certain amount (not even close to the full training set) the "converged to" loss got higher. It seemed as though the network was retaining it's accuracy on the portion of the dataset that it trained on earlier and all the new loss attributed to the latter data. I also tried training from scratch with the full dataset and it quickly converged to an unacceptable loss that performed poorly in all cases, not even just on the validation set but even on the training set

Intuition points to the network not having enough complexity to support the variation among the problem set. So I increased the complexity of the architecture and number of layers and repeating the process of testing the waters with increasing subsets of training data without re-initializing the weights.

Thank you for reading

  • $\begingroup$ At this point, a couple days later, the rate of change of loss is slow but still trending downward. I am not yet discounting the present architecture and continuing on with the intuition (hypothesis) that gradient descent steps increase with design complexity. Intuition being the error surface is a trickier terrain to navigate and hunt through. At least until I hear or observe otherwise. $\endgroup$ Commented Feb 9 at 1:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.