I believe that the resulting accuracy of your model is due to a mix of reasons. Firstly, the data-set is really small, so you cannot expect to have an efficient model just by using your 200 images.
Proposed Solution #1: Data Augmentation. Enlarge your data-set by processing your images, crop, rotate, etc Examples with code. As for labeling the bounding boxes, you could do it manually or by creating functions to transform the co-ordinates based on the image processing method.
Proposed Solution #2: Also, try a different approach. I think that a version of the Yolo algorithm would be more appropriate for multiple instance detection YOLO-read_the_docs_link because R-CNN's search algorithm is fixed, no learning happens during selection, which might lead to bad region proposals resulting to fewer detection and low probability outputs.
Why does the model detect objects more efficiently when zoomed in closer?
The obvious reason is that it is easier for your model to detect them, given the fact that the data-set is small. Also, it depends on the labeling, if your ground truth contains only large bounding boxes, then it would detect bigger pomegranates more efficiently. Adding cropped instances of your data-set to the train set might help with this issue too.