Context: my team is researching some plant-growth patterns using object detection models. We have collected a small set of images (a few thousand) and ran some simple experiments. The results were encouraging, so now we want to do some serious data collection (tens of thousands of images), but before doing so, we would like to have (create?) some "collection protocol", such as: defining the resolution of the images, times of day, weather, etc.

For example, since our application will only run during daylight, we won't collect nighttime images. And since we cannot control the weather, we want to ensure that we have training data with different conditions, such as rain, fog, etc.

Although (we think) our "collection protocol" is reasonable, it is still an ad-hoc product. We would feel better about doing such extensive data collection if we had some sort of "formal guideline", you know? I have read multiple papers discussing how datasets are made, but they usually focus on "how the data was labeled" and not on "how the data was collected".

Can you guys suggest some peer-reviewed papers that discuss the data collection process itself? We hope to avoid non-obvious pitfalls that we have not considered.

Thanks in advance.

  • 1
    $\begingroup$ Fwiw I think your thinking is good: the idea is to collect a representative sample of the data that the system will have to deal with in production. As much as possible it should represent the diversity of the production data and follow similar proportions. $\endgroup$
    – Erwan
    Feb 16, 2022 at 19:22

1 Answer 1


Some of tips that might be helpful:

  • Collect data incrementally, and run experiments meanwhile. Then update the annotation tasks according to model error analysis.

  • Design types of images collected based on use cases. In other words, image solution, time of day, etc. are dependent on the use case of ML model, and I think your analysis is on the right track.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.