Calculating Potential Usefulness of Acquiring Additional Data [closed]

Question

Imagine Anne has a labeled training dataset for a machine learning prediction problem. There is an opportunity to acquire more data from an agent, at a cost. However, before she decides to acquire that data by paying the cost, she wants to know if that additional data is likely to improve her model or not.

You can assume that there exists a black-box mechanism that allows Anne to perform some low cost computations on that additional data or the combined data (to explore the usefulness of that data). But she can NOT train a new machine learning model using the new data before she pays the non-refundable cost.

What kind of computations Anne should consider to get an idea/intuition of the added value this new data may bring? For example, if she could calculate a few metrics on the additional data or on the combined data, what should those metrics be?

How would your answer change if this was an unsupervised machine learning problem (e.g. clustering), and the datasets were unlabelled.

A few examples: Anne may be particularly interested in acquiring additional data to improve her model where it is weak. For e.g. this may be due to the fact that her original data may only cover a part of the feature space or distribution. Another example can be that her original data may have non-random missingness, which additional data may help with. It may also be useful to acquire more data points near the decision boundary etc.

I understand that the answers may vary depending on a lot of factors like the type of data, type of algorithm, the evaluation method, test distribution etc. But please feel free to make simplifying assumptions. The question is intentionally very general because I want to elicit answers from perspectives that I may not be aware of. You can also assume that Anne is indeed using the right model and the right learning algorithm, and there is scope to improve the model if she gets the right data.

Answer 1

How do I measure if new data would improve results?

In other words: How do I measure if new data will make more or fewer correlations in general?

A good metric is the entropy.

If you add new data and the overall data becomes more chaotic (= more entropy or less information gain), then you shouldn't take it into account.

If you have small samples, I don't know if they are enough to have a reliable entropy value, but it would be the same situation with most algorithms.

There are several ways to measure entropy: The Kullback-Leibler divergence, and the cross-entropy are good options.

Answer 2

You can make experiments using only a subset of the data you already have.

Suppose you machine learning should estimate some underlying unknown probability distribution from a given sample. If your probability distribution happens to be a Gaussian normal you only need to estimate the mean and variance but if the probability distribution is more complicated this problem can be a lot harder, especially if you don't have much a priori information on what the distribution could look like.

Of couse, the bigger your sample of training data, the better your estimation. But how much better would your estimate be if you could double your training data? Train your model with half or a quarter of the data you have and then try to get a feel for how much the model improves with each increase in the size of the training data.

Answer 3

You can use a Bayesian model to evaluate the confidence of your predictions. You can then use the confidence to specifically acquire more data where you model is less confident.

When training a normal / frequentistic machine learning model you only get one value back for every prediction you do (This is usually the mean or median of the Bayesian model). This does not incorporate the confidence of this value. Note, that in a classification problem you can't use the raw prediction value as a guide, because it does not represent any concept of confidence nor is the function linear or steady. I would also not use any normal evaluation metrics (mean squared error, area under the curve) as they are prone to overfitting. I guess the difference of these metrics between the training and test data could be an indicator, but again it does not represent any concept of confidence. A Bayesian machine learning models returns more information with which you can evaluate the confidence of the prediction.

Known and unknown unknowns

Ideally your Bayesian model distinguishes between known unknowns and unknown unknowns. Known unknowns lie in the non-determinism of your data. E.g. you can't predict the outcome of a coin throw and some dogs vs. cats are indistinguishable. Unknown unknowns come from the incompleteness of your knowledge. The model hasn't seen all examples, which are out there. There still could be some data out there which surprises your model.

If you are using Tensorflow you can use TensorFlow Probability for this. Here is great talk with an accompanied blog post to get you started on the topic. It also explains the concept of known and unknown unknown in more detail, but beware you probably need to watch/read it multiple times. For other machine learning frameworks similar extensions exist.

Bayesian bootstrap

A very cheap way to build Bayesian model is to use a Bayesian bootstrap (I wouldn't recommend the normal bootstrap method as it can produce bad results for small datasets). However, this only considers unknown unknowns. Here is a good talk about the usefulness of a Bayesian bootstrap in general. It's from the author of R package to do this. A similar package for Python exists, but it is not as well maintained, but the core bits for training a Bayesian machine learning model work. I checked them and contributed to the library a while ago. The code to do this is also very short so you can even implement it yourself. Basically you train many models (an ensemble) with different weights for the data points.

The output of this ensemble of frequentistic models is an ensemble of predictions. The closer the predictions are together (the smaller the variance) the more confident is the model about the prediction. With this technique you can also identify data points where the model is not very confident and especially ask for more example of those when acquiring more data. This is called active learning. You can listen to this podcast and explore this demo to learn more.

Some ideas

As for predicting how much the new data will improve you model, I have a few ideas, but have not tested them: You could train on smaller subsets of your data and plot the rising confidence levels over the size of the dataset. However, this would only work if you pull in new data randomly. When specifically choosing good new data points you would probably need to simulate this with your current dataset. There are probably many papers out there in the active learning space.

Answer 4

In my experience the most valuable data that Anne could have in addition is identified by asking validation questions "do we model the right data?" (as opposed to verification as in calculating some metrics: "does the model predict the existing data right"):

Are there factors/sub-domains of the data or model space/scenarios for application use that are not yet covered?
E.g., say, Anne trains some medical classifier and her training data was recorded at hospital X. Now hospitals Y and Z offer to join the project - this would be very valuable since Anne can now look at inter-hospital variation.

This recommendation boils down to putting together a list of potential influencing factors (including confounders) and trying to identify whether the new data offers additional coverage here.

---

An entirely different scenarios that looks at new data points within the factors already covered but also related to the question is deciding for which (existing) data points to acquire labels.

I have that when setting up regression models where many samples are measured (and all are to be predicted), but only a small number of the measured cases can be sent for (expensive, time-consuming) reference analysis. Kennard-Stone-algorithm or initially k-means can be used here for regression. This strategy is related to the Bayesian uncertainty in the other answer.

For classification, there is the additional question, whether the application requires more typical cases or more (deliberately) cases that are close to the class boundary - they may be acquired in a dedicated fashion. (This is again related to my first thought)