1
$\begingroup$

I am currently working on a product classification business case.

I have over 2 millions products from one source to classify in 23 categories. The input is the name of the product(ex:'Nike Air Max'). These products are already labeled. Whereas I can split my dataset into training and test set to compute the accuracy (which is 90%), I would like to predict categories of a whole new dataset, coming from an other source.

Once it will be done, I need to check products one by one to know if my classifier works well on this new dataset because it is not labeled and I really need to know the accuracy of prediction for this new dataset.

How can I know, the size of the sample coming from the new dataset (which has about 500k products) I need to check to be sure that my accuracy will have a small error margin?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The unlabeled data will have to be labeled to calculate accuracy. How much data to label is an empirical question.

Here are questions to answer to help guide that process:

  • How close is the distribution of the unlabeled data to the training data?
  • What is good enough for project goals?
  • What is small should error margin be? Can it differ by class?
  • Which is more important precision or recall?

Then monitor model performance. In particular, the confusion matrix and decision boundaries between important and commonly confused categories.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.