I am currently working on a product classification business case.
I have over 2 millions products from one source to classify in 23 categories. The input is the name of the product(ex:'Nike Air Max'). These products are already labeled. Whereas I can split my dataset into training and test set to compute the accuracy (which is 90%), I would like to predict categories of a whole new dataset, coming from an other source.
Once it will be done, I need to check products one by one to know if my classifier works well on this new dataset because it is not labeled and I really need to know the accuracy of prediction for this new dataset.
How can I know, the size of the sample coming from the new dataset (which has about 500k products) I need to check to be sure that my accuracy will have a small error margin?