5
$\begingroup$

I am working on classification of time series multivariate data. By doing PCA, I converted multivariate to uni-variate and fed it into a conv1d in keras.

However, I am getting a very high accuracy and low loss both in validation and in training. How can I justify this?

I have tried cross validation, but the results are not much different. I am using adam optimizer (learning rate:0.0001). With 0.001, my model fails to converge.

I have made sure that I am not mixing the training and validation datasets. I have shuffle both datasets independent of each other. I trained on 3728 samples and validated on 610 samples.

Can we expect such a high accuracy with binary classification?

Accuracy and loss curve

| improve this question | |
$\endgroup$
4
$\begingroup$

From the curves you are showing yes. Over-fitting would mean that your validation accuracy would be lower than your training accuracy, which is not the case here. Since you say that your training and validation sets are completely independent (i.e. no training samples are present in the validation set) you can consider the results reliable.

However accuracy might not be the best indicator of the model's performance. Make sure your dataset is balanced (i.e. the number of samples in both classes are equal to one another). If not try another metric that better represents the performance of your model.

| improve this answer | |
$\endgroup$
  • $\begingroup$ yes, my data is not balance, and i added class_weights = {0:1, 1:2}. I am even getting recall and precision close to 100%. Actually my work is a modified version of this paper Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals Once i replicated this paper on the same dataset the paper use, i modified the architecture, introduce some more pre-processing and tested the modified architecture on my data set $\endgroup$ – Talha Anwar Jun 22 '19 at 8:16
  • $\begingroup$ precision and recall is close to 100% for both classes? $\endgroup$ – Djib2011 Jun 22 '19 at 9:03
  • $\begingroup$ i did not check for individual class, my function is giving me average . let me check it for each class, then i will update it to you. $\endgroup$ – Talha Anwar Jun 22 '19 at 9:14
  • $\begingroup$ yes, for a particle split in cross val there is same precision, recall,for each class. Btw i do cross val and result is following accuracy: 0.9804497932991052 precision : 0.9843924777529307 recall : 0.9940812139070672 $\endgroup$ – Talha Anwar Jun 22 '19 at 9:58
  • $\begingroup$ If you get the same precision and recall for each class, the results seem reliable. I think your model is performing very well. $\endgroup$ – Djib2011 Jun 22 '19 at 10:40
0
$\begingroup$

Did you try keeping a separate test set (on top of train and validation sets) which is not used at any point in the model? Use this test set only after you have your final model, trained and validated. This should be a solid test of overfitting.

| improve this answer | |
$\endgroup$
-4
$\begingroup$

Over-fitting happens when validation accuracy begins to decrease while training accuracy continue to increase (and the opposite for the loss).
Thus looking at your curves, there is no over-fitting because validation accuracy never decreases significantly.
If training and validation losses have very similar values, it just means that your training and validation data are very similar i.e. are drawn from the exact same distribution, which is a good thing.
It is possible to have very high accuracy in a binary classification problem if the 2 classes are very different and so easily separable.
For instance you might get very high accuracy in recognizing cat versus dog but not so high accuracy in recognizing cat breed 1 versus cat breed 2 if the 2 breeds are visually similar.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.