# How to cross-validate a deep learning model for highly imbalanced datasets?

I am working with a multi-modality classification problem (with Keras). I have 1000, 5000 and 10000 samples for three different classes. I would like to do a five fold cross-validation to select the best pre-trained deep learning model for deployment. I'm including class weights during model training to give more weightage for less-pronounced classes. For a given fold, I would be validating with 200, 1000 and 2000 samples from these three classes. Is accuracy a good prediction measure to be used in this case? Or do I have to measure the F1-score and Matthews correlation coefficients as well? Am I doing it right?

Accuracy is not a good indicator of success with imbalanced data. The accepted answer is correct: F1 score is commonly used. Other options include roc_auc_score (see here) and average_precision_score (see here), both defined through scikit-learn.
If you're using Keras, I would recommend using class_weights (note this will not work well if you have a multi-label problem, although there are some workarounds, for example here).
You have imbalanced data set, so you should use F1 score. Also you can use weight for rare classes, so that your cost function will be formed in a way that it cares about rare classes so much and tries to classify them correctly. You also can use confusion matrix for the details, but F1 will suffice. And yes, use F1 instead of precision or recall. You can also take a look at here.