I have a question that my single deep neural network model gives above 90% accuracy for one data set and the same model gives an accuracy between 70-80% for the other data set. I want to know why this variation in accuracy happens in spite of the fact that i am using the same deep neural network model.
6
-
$\begingroup$ It happens because the data is different, of course. Are the two data sets equally sampled/distributed? If the model has seen one of the data sets during training, then the answer is overfitting, but there could be different reasons. $\endgroup$– MephyCommented Sep 5, 2018 at 13:09
-
$\begingroup$ Thanx Mephy for your valuable response $\endgroup$– Anam HabibCommented Sep 5, 2018 at 14:42
-
$\begingroup$ do you retrain the model on the new dataset? $\endgroup$– Mohammad AtharCommented Sep 5, 2018 at 19:38
-
$\begingroup$ Yes,i am using the same model on a new dataset. $\endgroup$– Anam HabibCommented Sep 6, 2018 at 2:52
-
$\begingroup$ I have a question. Are you training one model on one dataset or two models with same architecture with both the datasets? $\endgroup$– Mohit MotwaniCommented Sep 6, 2018 at 7:38
|
Show 1 more comment
2 Answers
$\begingroup$
$\endgroup$
3
I can think of one reason. Because you're using different data sets, the properties of your data set are likely to be different. So obviously what the model will learn will be different. And maybe the architecture of your network suits one data set more than the other. But also if you try training the model on your other data set(with less accuracy) for more epochs, it's possible that the accuracy will improve. Your model could be just be slow to understand this data set.
It's like when you see a human you can easily recognize a face. But when you see a software code, it's not always easy to find the bug. But if you're trained well enough you'll eventually be faster and better in recognizing the bugs.
I hope it makes sense to you.
answered Sep 5, 2018 at 11:30
-
$\begingroup$ Thanx @Mohit Motwani for your valuable response but i want to know that how should i know that what properties of dataset effects the accuracy of a model? $\endgroup$ Commented Sep 5, 2018 at 14:39
-
$\begingroup$ That's the thing about Neural Networks, it's difficult to understand what they are learning. For example, if you're doing an image classification task, you don't know what features it's using to distinguish the classes. So maybe you can try visualizing like in this paper cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf. There is no one method to do this(or that I'm aware of). You might have to do some trial and error depending on your application. $\endgroup$ Commented Sep 6, 2018 at 5:44
-
$\begingroup$
$\endgroup$
I'm assuming you're taking a certain neural net (NN) architecture and doing a train/test/validate cycle on dataset A to get 90% accuracy, then you reset your parameters and train/test/validate on dataset B to get 70% accuracy
To expand what @mohit-motwani says:
There does not exist a machine learning model that will approximate all functions to any desired degree. By corollary, there is no single neural net architecture that will best fit any given data.
Try tweaking various hyperparameters and see if that helps. Activation functions, convolution size, padding, pooling, even a few more/less hidden layers can drastically change your results.
NNs have been getting a lot of praise lately as "automated learning machines" but in reality, you still have to know the theory behind them, and what's under the hood to fully appreciate how they work. You'll get better with experience, but don't think any one NN architecture is "best"
