1
$\begingroup$

I have a Classifier which I have trained and tested on a small dataset - receiving solid results, though I wish to improve them. If I understand correctly one way of doing so is to add more data to obtain a more precise classification rule.

When doing this should I add data to both the training set and the test set? or should I add only to the training set? or maybe I should I create new training and test sets from the 'new dataset'? (new = the old data + the new data).

$\endgroup$
2
$\begingroup$

Adding more data does not always help. However, you can get an estimate if more data will help you by the following procedure: Make a plot. On the $x$-axis is the amount of training examples, starting at one example per class going to wherever you are currently. The $y$-axis shows the error. Now you should add two curves: Training and test error. For low $x$, the training error should be very low (almost 0) and the test error very high. With enough data, they should be "about the same". By plotting those curves you can make an educated guess how much more data will give you how much improvement.

When doing this should I add data to both the training set and the test set?

Depends on what you want to achieve. If only getting a better classifier, then you can only add it to the training set. However, if you're doing this in a scientific setting this might be more difficult. (I assume that your test set is of reasonable size).

You might want to have a look at cross-validation.

$\endgroup$
  • $\begingroup$ Thanks for the answer Martin. I am doing this in a scientific setting - what did you mean by "this might be more difficult"?. $\endgroup$ – Nimrodshn May 3 '16 at 7:22
  • $\begingroup$ @Nimrodshn The problem is that you want to compare what you do with other people. This is best if you work on a dataset on which other people are working, too. If there is no established dataset, you want to be at least consistent with youself and get models which you can compare with other models you had. (Assuming you do research in the direction "How can I solve a classification problem in domain X"). If you want a more substantial answer, you should probably post your research question / aim. $\endgroup$ – Martin Thoma May 3 '16 at 8:34
4
$\begingroup$

In order to improve your classifier, you have few options.

  • Ensembling - make a group of classifier and let them predict together. Stacking, blending, bagging, boosting. Choice is yours.
  • Hyper parameter tunning - you did not mention your tool, but i suppose that in every solid one is option to search in parameter space to find the best combination
  • Sampling - you can try undersample or create new samples (SMOTE) to give your classifier more data of class which you'd like to predict
  • Feature engineering - get rig of noisy features and work only with those which have an effect on your prediction, consider PCA too
  • Scaling - normalize your data can improve the performance, some classifiers require this
  • Data quality - missing values, coding categorical variables or suspicious values can influence performance
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.