I have a dataset from EEG data that is 24 features (24 electrodes) and 88000 samples with 3 classes, it is normalised and everything and had some noise filtered out via bandpassing.
When I classify with anything but a neural network the accuracy is pretty bad and I am using a 60/40 for training/test set just to make sure I can trust the result.
For example:
- Gaussian naive bayes: 42%
- Logistic Regression: 52%
- Linear Discriminant Analysis: 51%
However I played around with a neural network achieving 95%+ averaged with:
- 3 hidden layers: 100, 200, 100
- Activation: relu
- Learning rate: adaptive
I think this is super fishy so I did a PCA analysis
And plotted it with dimension reduction to two dimensions
As you can see, there is nothing significantly separable, which really confuses me. I am definitely using the test set to run the cross validation which is 40% of the sample data.
Can someone please advise on what's happening and whether I can trust this result? And what further steps I can do to make this result more concrete? I don't want to celebrate too early!!!