I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (GA), PSO, etc.). My training dataset is of size >=100,000.
Observation: For GA-trained DNN, I see that it obtains better accuracy than Adam-trained DNN but takes more iterations (hence longer time) than Adam. But only to perform slightly better, GA took 3000 iterations with a population size of 100 whereas Adam took only 100 epochs! I felt the performance gain does not sit right given the longer training time (even after using parallel processing for objective fn. computation). But before I conclude I want to know if understanding and implementation are correct as given below.
Implementation and problem: While implementing Adam with 100 epochs I use a batch size of 32. Whereas I don't how to use the concept of batch size with GA training but rather the usual initialize population and run some iterations style. Hence for each candidate of the population, the entire 100,000 samples are traversed through before loss (objective function) is computed. Then the same for the second candidate until 100 candidates are done. And this is just for 1 iteration! I am sure this is what increases the training time. But is there any other way to implement this kind of population-based optimizer here?
I feel if I find some time-reducing way I can more easily run these different population-based optimizers and experiments for comparison since I see a potential to obtain better predictions with these.
