New answers tagged nlp
0
Technically BOW includes all the methods where words are considered as a set, i.e. without taking order into account. Thus TFIDF belongs to BOW methods: TFIDF is a weighting scheme applied to words considered as a set. There can be many other options for weighting the words in a set.
Compared to regular TF-weighted BOW, the TFIDF weighting scheme gives more ...
2
It depends on the problem you are trying to solve. If you know the signal in the dataset already, the words which decide your decision then go with Bag of Words. This is useful when you are doing something like text classification.
On the other hand, TF-IDF is useful when you don't know the signal in the dataset. If you want to do text similarity, then, this ...
1
Glad you found where it went wrong! However, it is really possible for something like that to happen. There is no such thing as "best algorithm", so the performance of a method partly depends on what your dataset looks like. Or sometimes your feature engineering method just allows the data to cheat on you, say, you mistakenly leaked some data, or ...
0
I suggest hugging face is a good try although I have little experience using it. If it ultimately does not work well, you can also try NLTK, gensim and SpaCy. They are all widely used in NLP.
Here is a demo notebook that I found:
https://www.kaggle.com/thebrownviking20/topic-modelling-with-spacy-and-scikit-learn
Hope it helps and best luck to your ...
0
The recommendation is going to be still Huggingface transformers. You extract the features from BERT, then, some dense layers and then, feed them into sigmoid layer with unit equal to number of classes. Pose it as a multi-label classification.
0
Thanks to everyone who gave answer and comments. It was indeed caused by my data.
Prior to this I had the same preprocessing pipeline for both models, which would be your "usual" NLP preprocessing steps (non-alphanumerical removal, lowercasing, stemming, and stop word removal). I had a hunch that both stemming and stop word removal would cause the ...
1
Yes, this could be possible if your dev/test data comes from the same domain as the training data, in which case word2vec will encounter fewer OOV tokens that mess up the loss.
This could also mean that the benefits of BERT - subword tokenization to handle OOV characters in generalized domains - are lost. If your vocabulary size is small, your word2vec model ...
0
Although, the previous answer is a good reference to find how to measure probability of a sentence using BERT, in order to perform a meaningful evaluation of cross-model (e.g., compare BERT with Roberta) they should use the same tokenization.
3
LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing the number of topics, you can experiment by tuning hyper parameters like alpha and beta which will give you better distribution of topics.
The alpha controls ...
0
First there is a simple theoretical reason why relying on the probability provided by a NB model is not a good idea: what NB predicts is the posterior probability, i.e. the conditional probability $p(C_i|d)$ for every class $C_i$ for a given document $d$. The class $C$ which is predicted is just the one which obtains the highest probability $p(C_i|d)$. The ...
1
With regard to a dictionary of words, there can be no single dictionary for BERT because the BERT embeddings incorporate contextual information (i.e. the surrounding words in the sentence change the embedding for your target word). In theory, you could construct a dictionary for your words by passing single word sentences (though a single word may be broken ...
1
Translation as a pre-processing step is usually sufficient for many tasks (e.g. sentiment classification), but naturally undesirable for other tasks e.g. grading someone in written Dutch fluency.
Hence, for these tasks, the objective is:
Be able to train a language model for your specific language
However, you want to be able to do this with minimal ...
1
BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation.
You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-...
0
The evaluation design depends on the goal, i.e. how the classifier is intended to be used in production. The goal of a one-class classifier is to find instances of this class among other instances, so it has to be evaluated with data which includes negative instances. The choice of the proportion and type of negative instances should be driven by the final ...
1
Maybe this article will help you How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.
Talks about contextual word embeddings like BERT and GPT how they can capture various polysemous concepts rather than the static word embeddings which create a single representation for each word, such as GloVe.
0
What you are trying to address is a problem of hierarchical classification in contrast to the flat classification which we are very familiar with.
Some work has already been done to address such problems and it has been shown that single unified model outperforms layered architecure of multiple flat classifiers for individual tasks (e.g. in your case ...
1
You can get a dendrogram from any hierarchical clustering method. The tricky thing here is how to compute the distances between the words. If efficiency is your main concern, I would consider using HDBSCAN clustering.
The Jaro-Winkler distance was originally designed for such tasks. There is an efficient implementation in the python Levenshtein package, but ...
1
I should add, since you mention FastText, that FastText uses subword information to build its word vectors. Subword information is not tied to any specific word and can therefore be used to create vectors for OOV or rare words (the authors of the FastText algorithm specifically mention the ability to cater to rare word vectors not encountered).
BERT, GPT,etc ...
0
Simply you did not train the model.
Here you can find the documentation. However, just look at your function (and possibly put the import statements outside of it), you never use label variable.
You want to add the instruction
ml_model.fit(features, label)
before the return statement.
After that, you never call fit_transform on test data, only transform.
0
The key idea in both the papers are word embeddings are replaced by trainable activations that are computed using a nueral network. The network (referred to as bottleneck layer) takes word projections as input. The parameters of this network is shared across tokens. This results in a trainable token representation. The projection itself is not trainable, but ...
3
I don't understand how a pre-trained model can adapt to my given corpus
You are correct in thinking this way. It is not a magic wand. It learns the embedding values based on the underlying context of the corpus(e.g. news) which may work in the broad sense but not in a specific case.
Two cities may get the embeddings based on their geographical location but ...
1
The brain of a model resides in its weights.
Before any training happens - an empty model's weights are randomly initialized. The model training process then adjusts the weights into a more "favorable" region in N dimensional space.
So when you use pre-trained models - your model weights actually start from a "favorable" region (...
1
This question is quite open, but nonetheless, here are some:
lemmatization/stemming only makes sense in languages where there is a lemma/stem in the word. Some languages like Chinese have no morphological variations (apart from some arguable cases like the explicit plural 们), and therefore lemmatization and stemming are not applied in Chinese.
Word-based ...
0
To complement other answers:
You can featurize both sentences and then look at cosine similarity between their feature representations.
To featurize text, there are many methods you can use; from simple counting-based operators like TFIDF to word embeddings like word2vec or more complex language models like BERT.
The TextWiser Library might come in handy if ...
1
Those vector relations are not exact. Rest assured that king - queen ≠ man - woman. What we do is finding the closest vectors to the result of king - man + woman. One of the closest vectors is queen.
Nevertheless, when we try the "parallelogram approach" to verify word relations, in most cases, the closest vector is the original one. The fact that ...
0
Ok, I guess here how it should be read. A sample could be in multiple categories. First, you look into a sample. Below is an example (at row 12) and the mail content:
Content
Cat_1_level_1
Cat_1_level_2
Cat_1_weight
Cat_2_level_1
Cat_2_level_2
Cat_2_weight
Jennifer, Thank-you for stepping in on this and guiding the process! ---------------------- Forwarded ...
1
Firstly, I hope that the label is either a short summary or words of varying length, not just one word direction. Because moving cars involved in an accident may have multiple directions, or one car could be just parked like the example.
Secondly, given that you are planning to predict varying length label, and given the example text, I am pretty sure that ...
0
IMO. Firstly, you make the hypothesis that cliche sentences are in the same part of the document, which makes sense. But how do you determine the weight of the position and what if the students use different structures (e.g., some write the conclusion and the discussion sections separate, others write them together)?
Then you could get more up to date ...
1
The traditional approach for this kind of problem would be an n-gram language model. The language model is trained on a large corpus, then it's reasonably simple to calculate the most likely missing tokens for any incomplete sentence. SRILM was one of the most common toolkits, but there are probably many other libraries.
0
Methods for generating fuzzy (or soft) rather than crisp clusters may be applicable to your problem. One implementation for soft clustering in Python is a variant of DBSCAN called HDBSCAN. This link explains soft clustering using this algorithm. In soft clustering each point's similarity to each cluster can be measured. If a point is sufficiently similar to ...
1
the first few bits of the embedding are completely unusable by the
network because the position encoding will distort them a lot
This confused me very much at first because I was thinking of the model using a pre-trained word embedding. And then an arbitrary initial chunk of that embedding gets severely tampered with by the positional encoding.
However, in ...
0
For your first question, you can check if the tokenizer covers a certain string with the following:
text = 'today is a good day 😃'
ids2string = lambda ids: tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))
ids2string(tokenizer(text)['input_ids'])
> <s>today is a good day 😃</s>
If emoji is not included in the tokenizer ...
1
Looking at the first link from your post there is an in-depth overview of the different categories. Cat_#_level_1 denotes the top level category (12 in total) whereas Cat_#_level_2 denotes the second level category (up to 19 categories, depending on the level 1 category). E.g. an email with the labels Cat_3_level_1 and Cat_6_level_2 has the label california ...
0
You need to do pip install sentencepiece for it to work.
By the way, you can also give the tweets as a list to the tokenizer. You don't need to tokenize them one by one.
tokenizer(tweets, max_length=max_len, padding='max_length', add_special_tokens=True)
0
My question is — why does this token exist as input in all the
transformer blocks and is treated the same as the word / patches
tokens?
The CLASS token exists as input with a learnable embedding, prepended with the input patch embeddings and all of these are given as input to the first transformer layer. The CLASS token gathers information from all the ...
0
The transformers library uses complex output objects instead of plain tuples as return type since one of the updates after 3.5.1.:
from transformers import BertModel, BertTokenizer
t = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
o = t.encode_plus('this is a sample sentence', return_tensors='pt')
...
0
Disclaimer: this answer might be disappointing ;)
In general my advice would be to carefully analyze the errors that the model makes and try to make the model deal with these cases better. This can involve many different strategies depending on the task and the data. Here are a few general directions to consider:
Most of the time the imbalance is not the ...
0
(This answer was originally a comment)
You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...
Top 50 recent answers are included
