I have a fully-connected neural network with one hidden layer with 2 units which its goal is to classify a dataset with 324 samples and 300 features. 50% of the dataset is used for train and 50% of it is used for the test. There are two classes in the dataset which one of them has 75 samples and the other one has 249 samples. When I train the model with different seeds (actually when I re-run the training function), its results on the test set have a high variance, and the accuracy of the classification for test set changes about 20%. What is wrong with the model? How can I make it stable? How can I report the results of it?
One thing you can do is train your model N times and report the average and standard deviation of the accuracy.
Is the train/test split fixed? That means, do you use the same train set for every evaluation? And do you have a stratified split? That means the class imbalance of your data is also present in the train and test data. Since your data is imbalanced imagine you have a split where most of your train data consists of class 0, your model will learn to achieve a low train-loss with predicting 0 all the time, but you will have a high test-loss since you have way more samples of class 1 in your test set.
It probably helps to check your predictions. Fix seeds and stratify your splits.
It seems your model is Underfitting.
Change in data sequence changes its learning path and since learning is not complete(Underfit), this position becomes a random position based on the path.
Had the learning been completed, irrespective of the starting path. It will reach the same End position.
You can't rely on whatever accuracy you are getting.
Work is not done. Don't think about the reporting strategy now
Issue seems with high dimension data.
Imagine a space of 300 dimensions but only 150 points. Would be difficult for any model to learn.
What can be done -
If data is Sparse, try using SVM once.
if not, reduce the dimension and simply use RandomForest.