I am working with a text summarization problem, and I am trying to use this architecture [Pointer Generator]. My data set is VERY small (225 samples) compared to the CNN/ Daily Mail dataset this paper uses. I have decided to not go down the pretrained model+ fine tuning route and instead want to experiment with "co training" the model from scratch on CNN/Daily Mail data and my dataset since these two are structurally same. My dataset also involves condensing large text into one or two sentences, while introducing novel words not present in the input.
I am thinking of using 10% of my data and 90% of CNN/DM data in the first epoch and going down, use 100% of my data. However going down this route - I would only ever want to deal with 225 examples in an epoch. And incrementing the percentage of my dataset within this "combined" dataset by 10% in each epoch, I'd have a total of 11 epochs. I am confused if this is a feasible way to go about it. And if 225 examples in an epoch is a good number since the authors originally achieved results with a significantly large dataset.
Moreover, will it be okay if I do not show the model the same data from CNN/DM during consecutive epochs?
I am also looking to understand where in this code can I introduce the said changes?