0
$\begingroup$

Given that I have a machine learning model.

I evaluated the model over several labeled datasets and acquired the accuracy (or any other metrics) for these datasets.

Now I receive a new dataset without labels. I run the model and got the prediction.

Is there any way to estimate the accuracy of my model on the new dataset?

I was thinking of measuring the similarity of the datasets (in the feature space), then based on the similarity to predict the accuracy of the new dataset. For instance, if the new dataset is not far from the old dataset, the accuracies might be similar.

$\endgroup$
2
$\begingroup$

Is there any way to estimate the accuracy of my model on the new dataset?

No.

If you want to evaluate the classification accuracy of your model, you need to know the ground truth to compare, simple as that. Estimating the similarity of the datasets is totally irrelevant, unless there is prior knowledge that feature similarity leads to label similarity.

$\endgroup$
1
$\begingroup$

I have not heard about such an estimation. Think about it like this: You need a method, that given a similar dataset will predict labels for another dataset. Then, you will use the labels as the groundtruth. But how such a method is different from a machine learning model?

In other words, any estimated accuracy on such labels will tell you not how close is your model to the real annotations, but how is it close to another model used to generate the pseudo-groundtruth. This will make sense if your 'label generator' performs with a near 100% accuracy, but I am afraid such tasks are very rare. At least, outside labs.

$\endgroup$
0
$\begingroup$

I don't get it. You have data with labels and features. And are you using supervised machine learning algorithm. If yours new dataset hasn't labels, you have two options: - add labels to dataset - request for another dataset with labels

If you haven't labels you cannot use metrics like accuracy, precision etc because you don't know what's is TP or TN.

$\endgroup$
  • $\begingroup$ That's why my question is "to estimate", not "to measure". I am asking if there is a technique that estimates (or predicts if you like) the performance of the model if we perform the model on a particular dataset without the label. $\endgroup$ – mommomonthewind Jan 19 at 12:27
0
$\begingroup$

I'm not a machine learning expert by any means, but I feel like that's what the error measurement is on your training data. Or at least, in theory.

If it turns out that this isn't the case, then the data you used for training wasn't distributed in a way that represented the thing you're trying to model or you overfit your training data.

You can get a good "estimate" for the performance of your model by using a validation set, which basically helps you to determine if either of these two things have occurred.

$\endgroup$
0
$\begingroup$

Technically there is no direct way to do that. But let's us say you are doing some binary classification and the response rate in your training data set is 30% . Now when you use your model to score the new data set (without labels). Ideally average score should be around 30% (considering new data set is similar to training data set) .

In case of regression problems, the response rate can be replaced by average of dependent variable.

$\endgroup$
0
$\begingroup$

There is no direct way to estimate the accuracy. I would highly suggest Question on bias-variance tradeoff and means of optimization for how variance/bias is evaluated/defined in ML modelling, which can at least provide some context of how "accurate" (I am hesitant to use that word) should be defined here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.