I have a seq2seq conversational model (based on this implementation) trained on the Cornell movie dialogs. Now I want to fine-tune it on a much smaller dataset. The new data comes with the new words, and I want UNKs for as few new words as possible. So I'm going to create a new network with respect to the new input/output sizes, and I'm going to initialize its submatrices with learned weights I have at hand.
Could you say if this method can cause problems with the resulting model's performance? E.g. are the softmaxes likely to be affected significantly with these new initially untrained weights? And if it's OK, do you have some examples on how to do it with the least pain in tensorflow's seq2seq setup?