Mathematical algorithms to generate labels for training data

Question

What are the possible approaches when we need to train a model, but the training dataset is really small? (Assuming we have a lot of data, just not many data are labeled)

I know a library from Stanford: https://hazyresearch.github.io/snorkel/ that can generate training labels based on a some pre-determined experts rules. (Side question, anyone happen to know what's the underlining mathematics/statistics of this library?)

However, I am wondering in the scenarios when the snorkel packages can not be used, what are the approaches to label more data for training? Could Maximum Likelihood Estimator be used here? How would one implement such algorithm for labelling the data for training?

BTW, I am looking for a mathematical approach, not a brute-force one like using Amazon Mechanical Turk.

Thank you!

Answer 1

Assuming we have a lot of data, just not many data are labeled.

Try to tackle the problem first by using an unsupervised approach, following that, use the learned features for any downstream task.
see Deep Generative Models.

generate training labels based on some pre-determined experts rules

If you have some function that can generate the data distribution then why you bother yourself with such finding yet another function that approximate the former.
you could use this approach for evaluation purposes, e.g. compare different models regarding the problem being solved.

What are the approaches to label more data for training

you could aggregate (average) different models that tackle the same problem but with a different dataset.

Could Maximum Likelihood Estimator be used here.

As I said, we could maximize the likelihood of the data, giving high probability to samples that are very similar to the training data is very useful at exploiting the internal structure (learning the data manifold) that could potentially disentangle the factors that are responsible for generating the data which presumably part of them could be a direct signal for predicting the target y that you are trying to predict.

Answer 2

Some approaches when there is a small amount of labeled data and a large amount of unlabeled data:

Semi-supervised learning https://en.wikipedia.org/wiki/Semi-supervised_learning - mixtures of supervised algorithms on labeled data and unsupervised algorithms on unlabeled data. One of them (label propagation) is even implemented in scikit-learn http://scikit-learn.org/stable/modules/label_propagation.html

Active learning http://active-learning.net/ - algorithms that actively choose the data from which they learn, so they can achieve better performance using less labeled data.

These two approaches are complementary. Therefore, there are combinations of active + semi-supervised learning algorithms.