My general intution says yes, your model might get benefit from pre-trained weights.
But, in transfer learning, two can things happen -
- Positive transfer - The pretrained model on big dataset performs very well when trained on a new dataset
- Negative transfer - The pretrained model on big dataset has a performance decline when trained on a new dataset
I will suggest a small experiment -
make a small subset of your fish dataset of ~ 10% ~ 8K images having same class proportion as the original and train your model for both cases, with transfer learning and without transfer learning. This small experiment will quickly determine which method is the most successful in your case (Fish dataset and YOLOv4).
More detailed discussion on the same is here -