1) I am currently trying to set up a feedforward neural network with highly imbalanced classes (binary classification) in which the number of observations of class 1 is very low (and the class of interest to predict).
The dataset is very large, so in order to make the network better to predict class 1, I have downsampled the class 0 observations in the training set (ending up with a training set of approximately 600,000 observations).
Right now, I am estimating the neural network using 'accuracy' as the metric to optimize, while using early stopping to optimize the accuracy on an unbalanced validation set (monitor='val_acc').
My question is, therefore, whether I also should downsample class 1 observations in the validation set? The final test will naturally be unbalanced with a low number of class 1 observations.
Edit: My initial thought was to do use a function that takes the unbalanced classes into account, but the problem was that the dataset is too large, so I run into memory errors before I am able to run the neural network. After studying different approaches, I've read several papers suggesting to balance the classes (in which the problem of memory was also solved). My thought was that by balancing the classes, the accuracy function would make sense to use, but since I optimize the network directly on the validation within the model, I'm confused whether the validation set should also be balanced
2) Additionally, is there any common order on how to optimize a neural network (etc. first determine the number of hidden layers, number of neurons, and so)? A paper or similar arguing for its order of optimization would be very interesting (haven't managed to find one myself).
Thanks in advance!