2
$\begingroup$

Last 4-6 weeks, I have been learning and working for the first time on ML. Reading blogs, articles, documentations, etc. and practising. Have asked lot of questions here on Stack Overflow as well.

While I have got some amount of hands-on experience, but still got a very basic doubt (confusion) -- When I take my input data set with 1000 records, the model prediction accuracy is say 75%. When I keep 50000 records, the model accuracy is 65%.

1) Does that mean the model responds completely based on the i/p data being fed into?

2) If #1 is true, then in real-world where we don't have control on input data, how will the model work?

Ex. For suggesting products to a customer, the input data to the model would be the past customer buying experiences. As the quantity of input data increases, the prediction accuracy will increase or decrease?

Please let me know if I need to add further details to my question.

Thanks.

Edit - 1 - Below added frequency distribution of my input data:

enter image description here

Edit - 2 - Adding Confusion matrix and Classification report:

enter image description here enter image description here

$\endgroup$
  • $\begingroup$ It looks like your model overfits did you try to do a train/test split? $\endgroup$ – Robin Nicole Mar 18 '19 at 23:00
  • $\begingroup$ Thanks Robin. Yes, I've have a 75/25 split. Just out of curiosity, may I ask what hint made you think that the model overfits? ps. Added frequency distribution of my input data in the question. $\endgroup$ – ranit.b Mar 18 '19 at 23:07
  • $\begingroup$ So I guess it is your test accuracy which decreases. If your training accuracy keeps on increasing but your test accuracy decreases it meanss your model is overfitting. $\endgroup$ – Robin Nicole Mar 18 '19 at 23:15
  • $\begingroup$ Would you please consider moving this as a Comment to the original question? It works much better that way than a standalone answer. $\endgroup$ – I_Play_With_Data Mar 19 '19 at 11:45
  • $\begingroup$ Just added: Confusion Matrix and Classification Report. Do you think this is a case of class imbalance ? (where the output classes with less training data are never predicted) $\endgroup$ – ranit.b Mar 19 '19 at 12:38
3
$\begingroup$

To answer your first question, the accuracy of the model highly depends on the "quality" of the input data. Basically, your training data should represent the same scenario as that of the final model deployment environment.

There are two possible reasons why the scenario you mentioned is happening,

  1. When you added more data, maybe there is no good relationship between input features and label for the new examples. It is always said that less and clean data is better than large and messy data.

  2. If 49000 records added afterward are from the same set(i.e. have a good relationship between label and features) as that of 1000 before, there are again two possible reasons

    A. If accuracy on the train dataset is small along with test dataset. e.g. training accuracy is 70% and test accuracy is 65%, then you are underfitting data. Model is very complex and dataset is small in terms of the number of examples.

    B. If your training accuracy is near 100% and test accuracy is 65%, you are overfitting data. Model is complex, so you should go with some simple algorithm.

    NOTE* Since you haven't mentioned about training accuracy, it is difficult to say what out of the two above is happening.

Now coming to your second question about real-world deployment. There is something called model staleness over time which is basically the problem of reducing model accuracy over time. This is the article by a product manager at Google explaining the staleness problem and how it can be solved. This will answer your second question.

Let me know if something is not clear.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Many thanks, Sagar. Great piece of advice. Let me ponder my model from your viewpoint. And, will go through the article you posted bit later tonight. $\endgroup$ – ranit.b Mar 19 '19 at 12:34
0
$\begingroup$

There is a false myth that more data means better classification. The model also needs to build up in its complexity otherwise, the model is just overfitting on the data.

Taking only a few random samples from the data is the best strategy to train models, rather than inputting every bit of data that we can find.

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Thanks for sharing your thoughts. I guess, I've to understand the overfitting concept more properly. $\endgroup$ – ranit.b Mar 19 '19 at 12:35
  • 1
    $\begingroup$ Sorry for returning back late. You mean - I'll start with few random and equal number of samples from each class as input , and then try to train the model? But in that case the reality would be different, right (with so many varieties of inputs)? $\endgroup$ – ranit.b Apr 8 '19 at 17:16
  • $\begingroup$ What do you mean by varieties of inputs? $\endgroup$ – abunickabhi Apr 8 '19 at 17:22
  • $\begingroup$ See the data distribution, I have 41 distinct output classes. I meant, should we try with sample data considering only 5 classes with equal number of records in each? By "varieties of input" I actually meant the sample of different output classes (used as input for the model). \Sorry for the confusion. $\endgroup$ – ranit.b Apr 9 '19 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.