I have a seq2seq conversational model (based on this implementation) trained on the Cornell movie dialogs. Now I want to fine-tune it on a much smaller dataset. The new data comes with the new words, and I want UNKs for as few new words as possible. So I'm going to create a new network with respect to the new input/output sizes, and I'm going to initialize its submatrices with learned weights I have at hand.

Could you say if this method can cause problems with the resulting model's performance? E.g. are the softmaxes likely to be affected significantly with these new initially untrained weights? And if it's OK, do you have some examples on how to do it with the least pain in tensorflow's seq2seq setup?

up vote 1 down vote accepted

Its okay as long as the nerwork you are planning to create has the same number of layers and units i.e the dimensions of your network must be compatible with the weights that you are borrowing from the trained model. Also it would be better if you follow the second blog post of suriyadeepan practical seq2seq where he trains a conversation model on twitter chat. The code is much simpler and easier to understand, also it is on a smaller dataset, also he mentioned that the bot trained on cornell movie dialog corpus wasnt performing so well. Mainly to use the pre-trained weights all you have to do is load the model, create placeholders for the weights, assign thr weights from the loaded model to the placeholders and run a forward pass. This blog and this question might help you with this task

  • thanks for the information on Twitter! The thing is though, my new model will be bigger because of the extended vocabulary (i.e. input layer, and thus every other one). I am planning to manually assign trained weights to the "original" subregions of the model, but for the "extended" subregions, it will be random initialization. So my concern is, does this mixed-initialized model have a chance to succeed with re-training on these new data? – Igor Shalyminov Jan 12 '17 at 11:09
  • 1
    I dont think that it will succeed, best would be to create your vocabulary, train word embeddings and proceed to train a new model. Adding random weights to unassigned regions will not help as they have not been trained to minimize any loss. So you are better off training your own model. – Himanshu Rai Jan 12 '17 at 11:12
  • Indeed, I will train it on the new data anyway - I just want to reuse existing trained weights as a pretty decent (yet partial) initialization. Do you think it has a chance, or it's even worse than initializing everything from scratch?.. – Igor Shalyminov Jan 12 '17 at 14:41
  • Oh allright. In that case, it has a chance of training faster but again depends on the kind of data you are giving it. If your vocabulary is significantly different I would recommend training from scratch, because the weights are a reflection of the pattern in your data and if you use weights trained on quite a different dataset then you are introducing quite a bias into the model. Had the previous model been trained over a variety of data this approach would have a higher chance of working. I am not saying this wont work, since it seems transfer learning works decently enough, so go ahead and – Himanshu Rai Jan 12 '17 at 14:59
  • Yes, the vocabulary will grow pretty moderately, by 5% or so. The old model has been in training for a while, really don't want to throw it away:) Maybe also mixing some of the old dataset into the new one would help the model to not completely forget what it had learned before. – Igor Shalyminov Jan 12 '17 at 15:29

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.