I have a fully-connected neural network with one hidden layer with 2 units which its goal is to classify a dataset with 324 samples and 300 features. 50% of the dataset is used for train and 50% of it is used for the test. There are two classes in the dataset which one of them has 75 samples and the other one has 249 samples. When I train the model with different seeds (actually when I re-run the training function), its results on the test set have a high variance, and the accuracy of the classification for test set changes about 20%. What is wrong with the model? How can I make it stable? How can I report the results of it?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ It's hard to know without seeing your model, but an off the cuff guess would be that your learning rate may be off and the better test results were a lucky find of the gradient. $\endgroup$– Sean PayneCommented May 14, 2020 at 14:53
-
$\begingroup$ I changed the learning rate a lot. I think the problem is that there are a lot of local optima in the loss function. I don't know whether I should reduce the dimension of the input features with the PCA or I should focus on changing the loss function or other solutions. $\endgroup$– user137927Commented May 14, 2020 at 17:12
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
One thing you can do is train your model N times and report the average and standard deviation of the accuracy.
Is the train/test split fixed? That means, do you use the same train set for every evaluation? And do you have a stratified split? That means the class imbalance of your data is also present in the train and test data. Since your data is imbalanced imagine you have a split where most of your train data consists of class 0, your model will learn to achieve a low train-loss with predicting 0 all the time, but you will have a high test-loss since you have way more samples of class 1 in your test set.
It probably helps to check your predictions. Fix seeds and stratify your splits.
-
$\begingroup$ Yes, train and test are fixed. The imbalance is present in the train and test data. I think the problem is that there are a lot of local optima in the loss function. I don't know whether I should reduce the dimension of the input features with the PCA or I should focus on changing the loss function or other solutions. $\endgroup$ Commented May 14, 2020 at 17:10
$\begingroup$
$\endgroup$
3
It seems your model is Underfitting.
Change in data sequence changes its learning path and since learning is not complete(Underfit), this position becomes a random position based on the path.
Had the learning been completed, irrespective of the starting path. It will reach the same End position.
You can't rely on whatever accuracy you are getting.
Work is not done. Don't think about the reporting strategy now
Issue seems with high dimension data.
Imagine a space of 300 dimensions but only 150 points. Would be difficult for any model to learn.
What can be done -
If data is Sparse, try using SVM once.
if not, reduce the dimension and simply use RandomForest.
-
$\begingroup$ I tried SVM and the imbalanced classification problem happens. It predicts all the data in the train and test as one class! When I use PCA to reduce the dimension (one of my main problems is how to find the appropriate dimension for dimensionality reduction), it gets better, but the accuracy is not that good. $\endgroup$ Commented May 14, 2020 at 20:16
-
$\begingroup$ Have variance > 90%. Do DecisionTree Feature importance. Try getting more data $\endgroup$– 10xAICommented May 14, 2020 at 20:20
-