2
$\begingroup$

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing.

In my situation, I want to classify my dataset into 100 categories. The dataset is extremely imbalanced. The dataset size is about tens of millions

First of all, I run the model and got the prediction on the whole dataset.

Then, I sample 100 records per category(according to the prediction) and got a 10,000 test set.

Next, I labeled the ground truth of each record for the test set and calculate the precision, recall, f1 for each category and got F1-micro and F1-macro.

How to estimate the accuracy or other metrics on the whole dataset? Is it correct that I use the weighted sum of each category's precision(the weight is the proportion of prediction on the whole) to estimate?

Improve this question
$\endgroup$
1
  • $\begingroup$ Since the distribution of prediction category is not same as the distribution of real category, I guess the weighted approach does not work. Any one can help? $\endgroup$
    – ken wang
    Commented Jan 19, 2020 at 9:43

1 Answer 1

Reset to default
0
$\begingroup$

Accuracy has a specific meaning classification - the data points with predicted labels must exactly match actual labels over the total number of data points.

In order to calculate accuracy, you need the actual labels for each data point. If you do not have actual labels for a data point, those data points can not be used in the analysis.

Improve this answer
$\endgroup$
1
  • $\begingroup$ True, but determining the true labels for a random subset of the data will still allow you to estimate your method's accuracy/F1/precision/recall/etc. on the full data. Your confidence interval on the estimates will be wider than if you had all the labels, but there's no reason you can't provide an estimate of accuracy from a subset of the data. $\endgroup$
    – Nuclear Hoagie
    Commented Jun 12, 2020 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.