How can I check if a bigger training data set would improve my accuracy of my scikit classifier, is there a method or something?
- Split your data into train / hold out datasets.
- Train the model on a fraction of the training data (say 50%) and test on the holdout dataset.
- Train the model on a larger fraction of the training data (say 75%) and test on the holdout dataset.
It's important that you use the same holdout data for testing so you can perform a true test of accuracy.
Since you're doing classification, you should check that your data is balanced, and adjust if not (this may also improve your accuracy without needing larger training data).
The Validation Curve method (available on Scikit) plots the cross-validation score of your metric as you increase the number of training examples. If the model performance starts stagnating with the training examples of your original dataset, it may be a symptom that a bigger dataset will not improve your classifier's performance.
This also allows you to clearly observe the Bias vs Variance behaviour of your model.
As shown in the image below (source), you have a high bias (underfitting) when the both training and validation performances are clearly below your target. On the other side, you can overfit and cause your model to perform much better on the training dataset than in the validation, causing high variance (aka overfitting).
A well trained model will perform with a good Bias vs Variance trade-off, both performing near the desired target and performing evenly in both training and validation datasets.