6
$\begingroup$

1) I am currently trying to set up a feedforward neural network with highly imbalanced classes (binary classification) in which the number of observations of class 1 is very low (and the class of interest to predict).

The dataset is very large, so in order to make the network better to predict class 1, I have downsampled the class 0 observations in the training set (ending up with a training set of approximately 600,000 observations).

Right now, I am estimating the neural network using 'accuracy' as the metric to optimize, while using early stopping to optimize the accuracy on an unbalanced validation set (monitor='val_acc').

My question is, therefore, whether I also should downsample class 1 observations in the validation set? The final test will naturally be unbalanced with a low number of class 1 observations.

Edit: My initial thought was to do use a function that takes the unbalanced classes into account, but the problem was that the dataset is too large, so I run into memory errors before I am able to run the neural network. After studying different approaches, I've read several papers suggesting to balance the classes (in which the problem of memory was also solved). My thought was that by balancing the classes, the accuracy function would make sense to use, but since I optimize the network directly on the validation within the model, I'm confused whether the validation set should also be balanced

2) Additionally, is there any common order on how to optimize a neural network (etc. first determine the number of hidden layers, number of neurons, and so)? A paper or similar arguing for its order of optimization would be very interesting (haven't managed to find one myself).

Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)
Improve this answer
$\endgroup$
4
  • $\begingroup$ My initial thought was to do use a function that takes the unbalanced classes into account, but the problem was that the dataset is too large, so I run into memory errors before I am able to run the neural network. After studying different approaches, I've read several papers suggesting to balance the classes (in which the problem of memory was also solved). My thought was that by balancing the classes, the accuracy function would make sense to use, but since I optimize the network directly on the validation within the model, I'm confused whether the validation set should also be balanced. $\endgroup$
    – SorenA
    Apr 26, 2019 at 20:27
  • $\begingroup$ (I ran out of characters - So I just wanted to thank you for your answer and thoughts also!) $\endgroup$
    – SorenA
    Apr 26, 2019 at 20:28
  • $\begingroup$ As the optimization of the network is over the validation data if you want to make the classes balanced, obviously the train and the validation set should be balanced. But it is not necessary for the test set. However, you can just downsample your data (randomly select a subset of your data) and then use the mentioned loss functions which are robust against imbalanced data. $\endgroup$
    – Alireza Zolanvari
    Apr 27, 2019 at 4:52
  • $\begingroup$ Thank you again for the answer and verifying my consideration, I see that the downsample approach makes sense, and I think, I will try both and compare them on my dataset. The test set is finally evaluated on PR AUC compared to different model setups, so I'll keep this unbalanced as you also mention. $\endgroup$
    – SorenA
    Apr 27, 2019 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.