New answers tagged nlp
0
For this problem, you need to develop three models:
Model 1- for two main categories
Model 2- for sub-category A
Model 3- For sub-category B
So when you want to predict the result for an unseen data, first you use the Model 1, to find the main category. Based on the prediction and by using an if-else statement, you decide to perform another prediction using ...
1
Because BERT accepts the artificial assumption of independence between masked tokens, presumably because it makes the problem simpler and yet gave excellent results. This is not discussed by authors in the article or anywhere else to my knowledge.
Later works like XLNet have worked towards eliminating such an independence assumption, as well as other ...
2
There is at least one way:
Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
Test the transcripts for beign grammaticaly/syntacticaly correct.
This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken
1
I will start with an advice - just google "n gram language model" and you will find a lot of good detailed explanations.
With that being said I will give a short explanation about the "training phase" of n-gram language models (answer to question 2). The simplest way to build an N-gram language model strats with finding a big corpus - a ...
1
Your understanding is not correct.
The relevant information is described in the original paper in section 3.2.2:
The three sets of projection matrices you are referring to are $W^Q_i \in \mathbb{R}^{d_{model} \times d_k}$ for the Queries, $W^K_i \in \mathbb{R}^{d_{model}\times d_k}$ for the Keys and $W^V_i \in \mathbb{R}^{d_{model}\times d_v}$ for the Values....
2
OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, RoBERTa, Electra, etc.
The easiest way to access them all in a unified way is by means of the Transformers Python library by Huggingface. This library supports ...
1
Timing your get_max function on the largest list you provided gives a time of a little over 9 us which seems quite efficient. I have, however, written another function that does the same but takes a little more than 3 us, so almost 3x as fast. For smaller lists this speedup will become even greater (~10x speedup for lists of size 1 or 2).
doc = [(0, 0....
1
No - GPT-3 API is not currently public.
However once you get access, the documentation can be found at https://beta.openai.com/api-ref.
0
Those labels are not primarily for features, those labels are primarily for targets. Person, location, and event for targets for named-entity recognition (NER).
0
I am using now Tidytext, dplyer and widyr packages to find duplicates
I have a data frame contains product description named short_text, the column name is header, I add an index column, then tokenize the data with feature = sentences.
I calculated the Term Frequency and Inverse Document Frequency TF-IDF.
I calculated the cosine similarity using ...
1
Although there are several nearest neighbor tool as one mention by @oneextrafact. The problem with that tool is you need to index the external database for a logic operation like you mentioned that you want to build some logic using over date.
I will recommend you is to extract a document vector. Although there are several approaches to do that either by ...
0
Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized with a range of -1.0 to +1.0, but the word-embeddings are initialized with a mean of 0.0 and s.d. of embedding_dim ** -0.5 (0.044 for 512, 0.03125 for 1024).
So, ...
1
If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens.
Let's break it down with numbers:
At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},....
0
One of the workable approaches:
If you have sufficient data, and, you need to so this often, you can train your own custom NER model. Write a pipeline to match drugs and compare price and deliver a small app that solves the problem once and for all!
Custom Named Entity Recognition models helps you identify a named entity from a given chunk of text. There are ...
0
As far as I understand, we are not passing $3 \ x \ 10$ but $\text{maximum_sentence_size} \ x \ 10$. Actually it is in a sense static and you can not exceed this maximum sentence size. What happens if your sentence is smaller thane this size? You just pad with "padding vectors". And make sure that your model is not attending to those padding ...
0
The dot product is the magnitude of the projection of one vector $v$ to $w$, which is not a very good similarity measure, however:
$$
v \cdot w = \lVert v\rVert\lVert w\rVert Cos(\theta)
$$
Being $\theta$ the angle between the 2 vectors.
And:
$$
v \cdot w = \sum_{i=1}^n v_i \cdot w_i
$$
Being: $\lVert v\rVert = \sqrt{v \cdot v} $
With very simple math we can ...
1
The task you describe corresponds exactly to Named Entity Recognition (NER). This is a standard task for which there are many available libraries. NER is usually done with sequence labelling models such as Conditional Random Fields. Sequence labelling implies that the data is provided as a sequence which looks like this:
During <features ...> O
the ...
1
You could use clustering with a more basic similarity measure, for example cosine or even simply the proportion of words in common (e.g. Jaccard, overlap coefficient). This should gives you groups of sentences which are "quite similar" against each other, whereas sentences in different clusters are supposed to be very different. This way you would ...
0
The reference points to the word2vec library (source code).
It does not use normalised vectors during training (although it indeed uses the cosine similarity metric for semantic comparisons on already trained vectors).
The reasons for using only dot product instead of cosine similarity during training can be due to:
Dot product is a variation of cosine ...
1
What you are trying to do here is named-entity recognition. Namely, the task consists of classifying substrings into a set of named entities (i.e. person, location, etc.). From a more formal perspective, this is a sequence labeling task that classifies parts of a sequence.
This task can be approached in different ways:
gazetteers/string matching
regular ...
0
First, you are right that word co-occurrence graphs have been used in various applications of NLP. More precisely in applications related to the meaning of the words, typically for topic modelling or word sense induction. These applications follow the linguistic principle that words which occur in similar contexts tend to have a similar meaning. This is the ...
2
Generating artificial errors is generally risky in NLP, because it's difficult to make sure that the type and distribution of errors correspond exactly to real human errors. If the artificial errors diverge from real errors and a model is trained based on this data, the model will appear to have very good performance since it will rely on the patterns used ...
1
BERT needs to be fine-tuned to do what you want.
GPT-3 cannot be fine-tuned (even if you had access to the actual weights, fine-tuning it would be very expensive)
If you have enough data for fine-tuning, then per unit of compute (i.e. inference cost), you'll probably get much better performance out of BERT.
6
This is specified in the original Transformer paper, at the end of section 3.4:
Transcription:
3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension 𝑑model. We also use the usual learned linear transformation and softmax function to ...
0
Try to find a pattern in the sentences which are failing.
Use negspacy, find out words which are negative, replace them with opposite words. eg -
'not good' -> 'bad'
try using bert transformers pretrained sentiment classification models.
0
Apart from the above answer, what you can do is create your own rule based entity matcher - https://spacy.io/usage/rule-based-matching
POS(part of speech) tagging won't help here.
1
News sentences will have more unique tokens than normal conversations.
Conversations have more stop words than news articles.
I think you can use bert or normal wordvect classification to train a baseline model here.
I would play aroud the pipeline of fake news classifier and news-conversation classifier. like passing the text to news classifier first and ...
0
You can do the above task by making your own pattern for entity extraction- https://spacy.io/usage/rule-based-matching
0
Here's what I would have done.
extract raw data from the the pdfs.
use rule based NER from spacy(it's quite fast), but it will require some manual pattern making.
Also apply multiprocessing if you can.
https://spacy.io/usage/rule-based-matching
You can also write a document classification to segregate the templates and send the templates to appropriate code,...
0
prepare an excel sheet with columns as - sentence, color,material,fit ,style,sleeve_lenth, etc. Treat it as training data.
https://spacy.io/usage/spacy-101#training, use spacy to train a model where you feed the model - sentence ,tags and spantokens , something like - [{'red color shirt':[['color',(0,3)],['type',(10,14)]].
learn how to create a train the ...
0
It's true that the nltk book doesn't seem clear about this. Traditionally NER models are trained with Conditional Random Fields so I searched for "nltk crf" and found this SO question which points to this detailed example for NER.
To answer your questions:
nltk itself doesn't appear to propose a CRF model, the example above relies on an interface ...
2
This looks quite similar to Named Entity Recognition (NER), which is traditionally done with a sequence labeling model such as Conditional Random Fields. Normally NER is used when:
The list of possible entities is not predefined: the training data might contain "Mr James Smith" but the test data could contain "Mr John Doe". In other ...
1
You might need an encoder-decoder framework that can be implemented using two LSTM/GRU models. The Encoder takes the first phrase in, outputs the last (summary) hidden layer, it is taken by the first Decoder state, alongside BOS token, outputs the second token, uses it to predict the third, and so on until you reach the end of the (response) phrase.
0
I'm not sure I fully understand the question, but if you have handwritten text, so the word 'cat' can be written in many different ways, you could train an object detector like YOLO or Faster R-CNN to detect this word (e.g. on your data or OS dataset like ICDAR2015-FST) or even separate characters therein. If, on the other hand, yo want to identify unseen ...
2
Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence tagger and should behave better. Here is the issue with the details of its inception. You can train it if you have segmented sentence data.
Another option is using NLTK's sent_tokenize, which should give better results than Spacy's ...
2
There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components
To solve your problem, I see at least three ways to do it.
NTLK
NLTK allows you to add known abbreviations as exceptions. See this StackOverflow post.
Use a regular expression
Since your problem is that you have some example of dots that shouldn't ...
1
Depending on the representation of your sentences, you have different similarity metrics available. Some might be more suited to the representation you are using than others.
One of the most popular metrics is the cosine distance.
However, you have other available in the literature, such as:
Jaccard similarity
Sørensen–Dice coefficient
Tversky index
You ...
1
If you have a large corpus of text mapped with their respective topics, you can train a Siamese neural network where you have two inputs (one sentence and one topic) and output a similarity score based on whether they are related or not. This would require a good dataset with a good variety of similar and not similar pairs of (sentence, topic) to be ...
1
First off, there aren't, to my knowledge, models trained specifically to generate ngram embeddings. Although, it would be very easy to modify the word2vec algorithm to accommodate ngrams.
Now, what can you do?
You could compute the ngram embedding by summing up the individual word embeddings. Potentially, you could apply weights based on tfidf for instance, ...
3
I summarize your questions and then try to answer under each bullet point:
How to remove punctuation marks (e.g. # for hashtags which is used in social media)
The first goto is a regular expression that is used in data preprocessing very frequently. But if you are looking for all your punctuation to be removed from the text you can use one of these two ...
1
A [CLS] token is added to the beginning of the sentence, and a [SEP] token is added to the end.
These two special tokens have specific functions:
[CLS] is needed because it was used in the training loss of BERT to keep the first output position for a different purpose than the rest: it was used for the "next sentence prediction" loss, which ...
0
Which transformer version are you using? I had to pin mine to transformer == 3.5.1 to mitigate that problem, when the hugging face team updated their transformer to 4.0 things started to break.
Hope it helps
0
Absolutely, you could use bert pre trained model to suggest corrected sentences without annotated data as long as you are using a pre trained model. Here is a good repo that shows it
https://github.com/sunilchomal/GECwBERT
Hope it helps
2
Can anyone explain to me, what is more natural in English and why ?
This is a classic example of PP-attachment ambiguity (PP = prepositional phrase). For a full overview of the problem and some traditional approaches, check out this paper. Here I'll cover the basics.
The quick explanation is that the first analysis corresponds to the interpretation
Twain ( ...
0
Removing stop words or keeping them is an empirical question. The effect will vary based on corpus and task. In fact, the definition of stop words depends on the corpus and the task.
One approach would be to benchmark the effect of stop words with cross validation for the specific scenario.
2
Very interesting question.
Easy, but probably lazy answer
When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't remove them, and if it doesn't matter, it doesn't hurt to keep them in. Obviously, if you can, try with or without stopwords, and see what works best for your ...
0
There is not much theoretical work on the topic. The number of dimensions for an embedding is an empirical question depending on the corpus and the task.
The "GloVe: Global Vectors for Word Representation" paper by Pennington showed that there are large gains in accuracy by adding dimensions up to couple of hundred then diminishing returns.
The ...
0
I am also fairly new to this, but am working on the same type of problem where I have to predict the sentence based on previous data. Like you want giving X value predict y value.
So, basically you need to create a dataset as per your requirement and once your dataset is ready you can train your model with encoder-decoder or LSTM layers.
Please find below ...
0
All the tokens are inferred at once, independently from one another.
0
In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks:
Masked language modeling: some random words are masked with [MASK]
token, the model learns to predict those words during training. For
that task we need the [MASK] token.
Next sentence prediction: given
2 sentences, the model learns to predict if ...
Top 50 recent answers are included
