2
$\begingroup$

Let's say that we have a very small labelled dataset for instance segmentation and there is a photorealistic physics engine available that can produce synthetic data for us. Looking on the web I haven't found any paper about a similar issue.

Has anyone tried to use solely a photorealistic synthetic dataset or mixed with real data? If so, I would appreciate your conclusions or any relevant work that you can cite here.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I used such an approach at the Computer Vision Lab at TU Delft for my Bachelor thesis.

The goal was to analyze different aspects of the same problem: identifying LEGO bricks in a photograph of an unsorted pile of LEGO bricks. We built a small real dataset as well as a large (relatively) photorealistic synthetic dataset using Blender.

Personally, I looked at it from the explainable artificial intelligence (XAI) point of view, but some of my team mates did have objectives closer to yours (image segmentation, object detection, classification).

You can find a description which is a bit more detailed here, in section 4.1.

I can edit this answer with more details if you're interested.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your answer. Are you aware of any of your colleagues work if it has been published w.r.t to the objective of image segmentation or detection? Also, from the XAI perspective did you notice if the network was paying more attention to the artificial/real samples? $\endgroup$
    – Dimimal13
    Feb 7, 2021 at 14:55
  • 1
    $\begingroup$ This one explores object detection, it has a bit more on the topic of the synthetic dataset, the main take-away is that the performance on the synthetic dataset is worse than on the real dataset, but cropping the synthetic images improves it. $\endgroup$
    – David Cian
    Feb 7, 2021 at 16:22
  • 1
    $\begingroup$ This one explores image classification, might also be interesting. I didn't notice any major difference from the XAI perspective, except that real backgrounds make the task considerably harder for a model whereas our synthetic datasets where we just used flat images of backgrounds didn't present this problem. $\endgroup$
    – David Cian
    Feb 7, 2021 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.