Your hypothesis about missing colours in your samples affecting results in production could be correct. However, it is trivial to convert images to greyscale as you load them from storage. So keep them in colour, and convert them as you load them if you need black and white.
That way you can try both with and without colour as input and you will have your answer.
This is very little effort to do in practice, and allows you to do the "science" part of data science by comparing two approaches and measuring the difference. This is standard practice, even if you are reasonably certain one approach or another is "the best", it is normal to explore a few variations. When you are not sure, then it is even more important to try it and see.
To test your hypothesis, you could put all the t-shirts of a particular colour in your test set. If that reduces the accuracy of your results with the colour model, it would back up your concern. One fix for that might be to remove colour information from the model, if it is not relevant to the task. An alternative is to collect more data so you have enough samples of different colour shirts. However, you might find if you are fine-tuning a neural network trained on many more images (e.g. Inception v5) that the impact of colour is less even though your samples do not cover all possible T-shirt colours.