I am training the diffusion model from diffusion policy, specifically their vision notebook, on a custom dataset. As always, I try to make a sanity check of the pipeline, by overfitting on a single batch. I would expect the loss to go to 0 or nearly 0.
But the loss get's stuck pretty high. Interestingly, the results, when training on the whole dataset, still look okay. Also, when I train on a single batch but fix the diffusion iteration, the loss does decrease nicely to 0.
Does somebody have experience with that? Is it not common to use this single batch overfitting for diffusion models? Is this a sign of bad implementation?
Also, I should add, that I did the same with the notebook from diffusion policy. I only trained their network on their data on only one batch. And it also did not go nicely to 0.
DiffusionPolicy: https://github.com/real-stanford/diffusion_policy?tab=readme-ov-file Vision Notebook: https://colab.research.google.com/drive/18GIHeOQ5DyjMN8iIRZL2EKZ0745NLIpg?usp=sharing
