Pretraining a neural network to teach it general information

Question

Let's say that I want to train a neural network to recognize symptoms of severe dehydration in distance runners visually. Runners tend to finish races either overhydrated (hyponatremic) or dehydrated, but the two present with fairly similar symptoms, so knowing the difference could be quite useful.

I've got myself a small dataset of a few thousand faces of dehydrated runners, a few thousand of overhydrated runners, and several million faces of the general population. I don't expect to be able to properly train a network to recognize dehydration with just a few thousand faces of the dehydrated, but I'd like to see if I can pretrain the network using the several million faces so it learns to distinguish facial details, then train it on the actual overhydrated/dehydrated faces. But if I take some other problem, like distinguishing men from women, and pretrain it on that, I'd imagine my hydration training phase would output something that distinguishes something between hydration level and gender.

Is there some standard way to pretrain a model to learn general details (of faces for example) before training on the proper (far smaller) dataset?

Answer 1

Considering two tasks, first, classifying the gender and second, whether its overhydration or dehydration. We can't create a model that perform these two tasks in a combined manner. Yes, a system could be developed for the same.

Discussing pre-trained models for facial recognition and related tasks, we have the FaceNet model as well as VGGFace. We can build a model on top of these highly trained models.

1. A pretrained model as discussed above will first, provide a bounding box for each face present in the image. The image could be a video frame from a real-time stream.
2. We will crop that part of the image, so we are only left with a smaller image which only contains the face of the athlete.
3. Another model, which will be trained on overhydrated vs hydrated faces, will classify the face into the two classes as needed.

For each step, we can train a model and place it in the pipeline. Also, try weight quantization for a faster inference speed.

Answer 2

This looks like a classical case of Transfer_learning https://en.wikipedia.org/wiki/Transfer_learning. You have some powerful model ResNet, EfficientNet pretrained on large dataset - ImageNet, for instance.

Depsite solving a different problem, during the training convolutional layers are learned to extract useful features and patterns on the image. They achieve some more interpretable and convenient representation of data for classifier or detection layers, depending on the task.

However, the dataset, which provides one with the pretrained network,should be as close as possible to the target data. In case they have a few in common, the extracted features won't be of much use for the problem under interest, and in fact the whole network needs to be trained as from scratch.

For your problem the other answer proposes networks, trained on faces, which seems a reasonable choice.

Another thing, that may significantly improve quality for small data is a clever data augmentation.