I want to make a multilabel image classification model that can detect many different labels. For each label, I can get at least 5000 positive examples and 5000 negative examples. However, my question is how can I use data in this format to train a multilabel image classifier. Part of the challenge is, for example, I can download 10,000 images of a hand and know that they are positive examples but then if I want to detect a shoe as well, I don’t know how many of those hand photos might have also had a shoe in them. I’m trying to make a model this way because I will have a fairly high amount of labels and need to be continuously adding new labels. What is the best way to go about doing this?
Why don't you give Multi-label classification a try.you can actually train your model to predict multiple labels for one image by training the model on a dataset that has images with multiple labels the only problem you might face here is to where to get the data from you can actually use this dataset for practice: https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/data.
Another better option that you have is as you said in your question you want to train your model on multiple classes you can actually use Imagenet dataset it mas 1000 classes and more than a million images. Note:Use tranfer learning to build your model and do watch lesson 1,2,3 from fast.ai part1 2019 course you will have all your answers. Jeremy Howard is a god!!!!
I agree with khwaja wisal : having multiple binary datasets should be enough to train a multi-label model.
You can train a model on a noisy dataset and still get a decent model. Your dataset would be noisy because as you pointed out some images might include more than one label whereas you will only train it to detect the single label you actually have.
If you don't like the idea of training on a noisy dataset you can use some masked loss : use a multi label model to predict all labels but for each image only optimize the label of that image.
That means that your masked loss would be something like
multilabel_loss(y_true, y_pred) * is_label_known with label_known being a tensor with 1 if the label is known (positive or negative) and 0 if you have no idea if the label is there or not