Hot answers tagged

3

I don't understand how a pre-trained model can adapt to my given corpus You are correct in thinking this way. It is not a magic wand. It learns the embedding values based on the underlying context of the corpus(e.g. news) which may work in the broad sense but not in a specific case. Two cities may get the embeddings based on their geographical location but ...


3

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing the number of topics, you can experiment by tuning hyper parameters like alpha and beta which will give you better distribution of topics. The alpha controls ...


2

It depends on the problem you are trying to solve. If you know the signal in the dataset already, the words which decide your decision then go with Bag of Words. This is useful when you are doing something like text classification. On the other hand, TF-IDF is useful when you don't know the signal in the dataset. If you want to do text similarity, then, this ...


1

Glad you found where it went wrong! However, it is really possible for something like that to happen. There is no such thing as "best algorithm", so the performance of a method partly depends on what your dataset looks like. Or sometimes your feature engineering method just allows the data to cheat on you, say, you mistakenly leaked some data, or ...


1

Yes, this could be possible if your dev/test data comes from the same domain as the training data, in which case word2vec will encounter fewer OOV tokens that mess up the loss. This could also mean that the benefits of BERT - subword tokenization to handle OOV characters in generalized domains - are lost. If your vocabulary size is small, your word2vec model ...


1

Translation as a pre-processing step is usually sufficient for many tasks (e.g. sentiment classification), but naturally undesirable for other tasks e.g. grading someone in written Dutch fluency. Hence, for these tasks, the objective is: Be able to train a language model for your specific language However, you want to be able to do this with minimal ...


1

With regard to a dictionary of words, there can be no single dictionary for BERT because the BERT embeddings incorporate contextual information (i.e. the surrounding words in the sentence change the embedding for your target word). In theory, you could construct a dictionary for your words by passing single word sentences (though a single word may be broken ...


1

BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation. You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-...


1

Maybe this article will help you How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. Talks about contextual word embeddings like BERT and GPT how they can capture various polysemous concepts rather than the static word embeddings which create a single representation for each word, such as GloVe.


1

You can get a dendrogram from any hierarchical clustering method. The tricky thing here is how to compute the distances between the words. If efficiency is your main concern, I would consider using HDBSCAN clustering. The Jaro-Winkler distance was originally designed for such tasks. There is an efficient implementation in the python Levenshtein package, but ...


1

I should add, since you mention FastText, that FastText uses subword information to build its word vectors. Subword information is not tied to any specific word and can therefore be used to create vectors for OOV or rare words (the authors of the FastText algorithm specifically mention the ability to cater to rare word vectors not encountered). BERT, GPT,etc ...


1

The brain of a model resides in its weights. Before any training happens - an empty model's weights are randomly initialized. The model training process then adjusts the weights into a more "favorable" region in N dimensional space. So when you use pre-trained models - your model weights actually start from a "favorable" region (...


1

This question is quite open, but nonetheless, here are some: lemmatization/stemming only makes sense in languages where there is a lemma/stem in the word. Some languages like Chinese have no morphological variations (apart from some arguable cases like the explicit plural 们), and therefore lemmatization and stemming are not applied in Chinese. Word-based ...


1

Those vector relations are not exact. Rest assured that king - queen ≠ man - woman. What we do is finding the closest vectors to the result of king - man + woman. One of the closest vectors is queen. Nevertheless, when we try the "parallelogram approach" to verify word relations, in most cases, the closest vector is the original one. The fact that ...


1

The traditional approach for this kind of problem would be an n-gram language model. The language model is trained on a large corpus, then it's reasonably simple to calculate the most likely missing tokens for any incomplete sentence. SRILM was one of the most common toolkits, but there are probably many other libraries.


1

Looking at the first link from your post there is an in-depth overview of the different categories. Cat_#_level_1 denotes the top level category (12 in total) whereas Cat_#_level_2 denotes the second level category (up to 19 categories, depending on the level 1 category). E.g. an email with the labels Cat_3_level_1 and Cat_6_level_2 has the label california ...


1

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot This confused me very much at first because I was thinking of the model using a pre-trained word embedding. And then an arbitrary initial chunk of that embedding gets severely tampered with by the positional encoding. However, in ...


1

Firstly, I hope that the label is either a short summary or words of varying length, not just one word direction. Because moving cars involved in an accident may have multiple directions, or one car could be just parked like the example. Secondly, given that you are planning to predict varying length label, and given the example text, I am pretty sure that ...


Only top voted, non community-wiki answers of a minimum length are eligible