# Loss decreases, but Validation Loss just fluctuates

I've been trying to implement object detection using a CNN architecture like this:

model = keras.Sequential([
keras.layers.Input(shape=(320, 320, 1)),
keras.layers.MaxPool2D((2, 2), strides=2),
keras.layers.MaxPool2D((2, 2), strides=2),
keras.layers.MaxPool2D((2, 2), strides=2),
keras.layers.MaxPool2D((2, 2), strides=2),
keras.layers.MaxPool2D((2, 2), strides=2),
]);



However, while the loss seems to decrease nicely, the validation loss only fluctuates around 300. Loss vs Val Loss

This model is trained on a dataset of 250 images, where 200 are actually used for training while 50 are used for cross-validation. Why could this be? Could my model be too deep? Do I need to reduce my learning rate even more? Or do I just not have enough data?

For reference I am trying to mimic the Tiny YoloV2 architecture shown here