I am a newbie in the field of deep learning, so advance apologies if any mistake is there. I was trying to use pretrained models like BERT and GPT2 for generating language in our native language (say, Hindi). These models are pretrained on English data corpus. When I fine-tune them on Hindi data, they produce reasonable Hindi sentences, but when I test with English words, they are producing something meaningless. It means that fine-tuning is completely removing the properties of the pretrained model. If it happens then what is the use of transfer learning? Why is this happening and is there any way of preserving the properties of base model as well as get task specific results?