New answers tagged

0

Thanks to everyone who gave answer and comments. It was indeed caused by my data. Prior to this I had the same preprocessing pipeline for both models, which would be your "usual" NLP preprocessing steps (non-alphanumerical removal, lowercasing, stemming, and stop word removal). I had a hunch that both stemming and stop word removal would cause the ...


0

Yes, this could be possible if your dev/test data comes from the same domain as the training data, in which case word2vec will encounter fewer OOV tokens that mess up the loss. This could also mean that the benefits of BERT - subword tokenization to handle OOV characters in generalized domains - are lost. If your vocabulary size is small, your word2vec model ...


6

The reason to average the embedded vectors of the words in a paragraph or document is to obtain a single fixed-size vector that represents the whole text. Then, the document-level vector can be used as input to a document classification model or any other document-level model. If you explicitly want to compute word-level representations and then combine them ...


1

I should add, since you mention FastText, that FastText uses subword information to build its word vectors. Subword information is not tied to any specific word and can therefore be used to create vectors for OOV or rare words (the authors of the FastText algorithm specifically mention the ability to cater to rare word vectors not encountered). BERT, GPT,etc ...


3

I don't understand how a pre-trained model can adapt to my given corpus You are correct in thinking this way. It is not a magic wand. It learns the embedding values based on the underlying context of the corpus(e.g. news) which may work in the broad sense but not in a specific case. Two cities may get the embeddings based on their geographical location but ...


1

The brain of a model resides in its weights. Before any training happens - an empty model's weights are randomly initialized. The model training process then adjusts the weights into a more "favorable" region in N dimensional space. So when you use pre-trained models - your model weights actually start from a "favorable" region (...


1

Those vector relations are not exact. Rest assured that king - queen ≠ man - woman. What we do is finding the closest vectors to the result of king - man + woman. One of the closest vectors is queen. Nevertheless, when we try the "parallelogram approach" to verify word relations, in most cases, the closest vector is the original one. The fact that ...


Top 50 recent answers are included