My goal is to build a statistical model with domain specific phrase embeddings.

To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence embeddings, I am more interested in complete phrases as embeddings. The goal is to be able to create this model and then use some kind of transfer learning to create phrase embeddings from the unsupervised task of skip-thought modeling.

One of the biggest issues we have is that while our corpus is fairly large, it is only 23,000 free text notes and thus, approaching maybe a 11 million words at best. Thus, a standard word embedding model would have issues wrt data sparsity. Also, our entities of interest are basically multi-token phrases. This is why we are looking at building a model using skip-thought vectors, where instead of sentence embeddings, the unit of interest is a phrase, and then using transfer learning to create phrase embeddings.

I am wondering if anyone is familiar with any relevant research on this, and how I would go about training a corpus of relevant documents to create a statistical model of phrase embeddings using skip-thoughts. Would I first create a word2vec model and then use that for input to further train the model? Is this similar to creating a word2vec skip-gram model wwithout worrying about sentence boundaries? Any recommended packages out there? I have not had any luck pinning down specifics on how to do this.

NB: One problem with our text, is that not all sentences in the text are clearly delineated as a sentence. So, I'm thinking this might be beneficial, in that I could have a boat load of skip-grams as phrases that cross sentences boundaries.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.