Estimating and/or determining the amount of data sufficient enough to train a model

Question

To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of perspectives can we try to estimate or/and determine how much data we should annotate? Or what kind of techniques can we use to predict the amount before we get the results of the first attempt?

I know that words can be separated into two types: closed class type and open class type. Should we consider that?

If such methods do not exist, how can we know if our model (mostly high variance) suffers from data shortage (knowing that data is never enough) or other reasons (like poor structure or bad training process or bad hyperparameters, etc.) after we get the results of our first version of the model and think the results are not satisfactory?

Answer 1

There are a few different perspectives from which to determine the amount of data you need. Those include:

• Project complexity: Each parameter that your model has to consider in order to perform its task increases the amount of data that it will need for training.
• Training method: As your models is forced to understand a greater number of interlinking parameters, the resulting complexity forces a change in the way that it is trained.
• Labeling needs: Depending on the task you’re doing, data points can be annotated in different ways.
• Diversity of input: In cases where your model’s input won’t be highly controlled, more data will be necessary to help your model function in that unpredictable environment.

You could begin working on your model with the data you have, then add more data when you feel it becomes necessary. Ideally, this will give you a very clear picture of your data needs, and they'll likely fit into one of the above categories.

This article covers training data needs in-depth, so it might be helpful. It also includes examples of dataset sizes for a variety of language-based and NLP projects. Unfortunately, there's not a lot of consistency among projects; some of the projects (text language identification) used 3000 training and 1000 test samples, while others (sentiment analysis) used datasets with over 10,000 samples. The differences here can likely be attributed to availability of data (tweet data is much easier to collect than South African language texts, for example).