I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a different enviroment and I want to embed the two words together, or the tree words together in their own envoroment.
Is this possible?
If yes how exactly?
Thank You in advance.
There are several ways word-embeddings are trained, however most of them require a ton of data. They usually involve learning vector representations that are useful for some self-supervised objective, which all tend to be pretty data-hungry.
If you have a large enough dataset you could train new domain-specific embeddings from scratch, but it probably be more effective and much easier to finetune existing embeddings (i.e., initialize on models/representations and train on your domain data).
See: this post for finetuning word2vec, for example.
If you want to train on phrases, then you will have to devise your tokenizer that way. Pseudo code will look like:
1. Write tokenizer that maps phrases to a single token
2. Using tokenizer generate vocabulary from text corpus
a. this will assign multi word phrase a single token. for example the phrase "Natural language Processing" will be considered as one token.
3. Once you have devised tokenizer that way, the remaining process will look same.
For example if you are training using gensim, Following line will use your custom tokenizer instead of default gensim function.
yield utils.simple_preprocess(line)
