# Tag Info

20

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this: The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even a human translator will not necessarily score 1. It is important to note that ...

8

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT. They also have a github repo which is easy to work with.

8

Other options would be to... Compare similar text sequences, Compare similar string sequences, Use fuzzy matching. Fuzzy Matching: library(fuzzyjoin) # https://stackoverflow.com/questions/26405895/how-can-i-match-fuzzy-match-strings-from-two-datasets a <- data.frame(name = c('Ace Co', 'Bayes', 'asd', 'Bcy', 'Baes', 'Bays'), price = c(...

7

Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two? There is a confusion here: NLP is the whole domain of AI which deals with natural language. It includes virtually any task related to processing language data (usually mostly written data, but that's not the point). Topic modeling is ...

7

The reason you're seeing BERT and its derivatives as benchmarks is probably because it is newer than the other models mentioned and shows state-of-the-art performance on many NLP tasks. Thus, when researchers publish new models they normally want to compare them to the current leading models out there (i.e BERT). I don't know if there has been a study on the ...

7

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes an autoregressive factorization of the probability of the sentence: $p(s) = \prod_t P(w_t | w_{<t})$ On the other hand, BERT's masked LM loss focuses on ...

5

The first line of the BERT abstract is We introduce a new language representation model called BERT. The key phrase here is "language representation model". The purpose of BERT and other natural language processing models like Word2Vec is to provide a vector representation of words, so that the vectors can be used as input to neural networks for other ...

5

It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that ...

5

I will take as reference fairseq's implementation of the Transformer model. With this assumption: In the transformer, masks are used for two purposes: Padding: in the multi-head attention, the padding tokens are explicitly ignored by masking them. This corresponds to parameter key_padding_mask. Self-attention causality: in the multi-head attention blocks ...

5

BLEU scores are based on comparing the translation to evaluate against a gold-standard translation. In general the gold-standard translation is the same source sentence translated by a professional translator, so in theory a professional human translation should always receive the maximum score of 1 (BLEU scores are normalized between 0 and 1). However it's ...

4

For pretrained models, spaCy has a few in different languages. You can find them in their official documentation https://spacy.io/models The available models are: English German French Spanish Portuguese Italian Dutch Greek Multi-language If you want support for extra labels in NER, you could train a model in your own dataset. Again, this is ...

4

In my opinion there are two ways: Ask a few experts to assess the quality of the clusters based on a sample (after the clustering has been done, much easier than pre-annotating the whole data especially in the case of clustering) If the clustering is done in the perspective of using the result in another task, the performance of this other task will reflect ...

4

2 approaches to correct misspellings: Make your own dictionary of corrections, for example: mispell_dict = {'colour': 'color', 'centre': 'center', 'favourite': 'favorite', 'travelling': 'traveling', 'counselling': 'counseling', 'theatre': 'theater', 'cancelled': 'canceled', 'labour': 'labor', 'organisation': 'organization', 'wwii': 'world war 2', 'citicise'...

4

BERT generates contextualized word embeddings, which means that BERTprovides the most accurate embeddings when a word is in a sentence(context). For each of the words within the sentence, BERT will generate a vector of numbers. In your case, you will have a good representation of the word "bank". So if you have a sentence for all the other words that you ...

4

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. ...

4

A way to speed up this process is to preprocess the large dataset, the goal being to store the documents from A in a way which avoids a lot of useless comparisons. Store each document from A in an inverted index $m$, so that for any word $w$ $m[w]$ is the list of all documents in A which contain word $w$ (note that a document can appear several times in ...

4

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it. The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-...

4

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included. Paper with code contains a section on NLP. That's a great website ...

4

You need character embeddings. I assume you are already familiar with word2vec technology. Its goal it to make a model "learn" the relative meaning of words, placing them into a highly dimensional space. The same can be done with single characters, instead of whole words. The preprocessing steps you need will be a little bit different, but the ...

3

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...

3

You might be interested in the open-source R package RBERT: https://github.com/jonathanbratt/RBERT It's a work in progress, but the goal is to be able to use BERT directly in R.

3

from keras.preprocessing import Tokenizer samples = ["grss is green and sun is hot"] tokenizer = Tokenizer(num_words=1000) tokenizer.fit_on_texts(samples) sequences = tokenizer.texts_to_sequences(samples) The Keras library uses it's tokenizer function but you have other well known libraries like nltk, gensim to convert them into sequences and pass it into ...

3

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it. However, working without the stopwords in your documents will make the number of features ...

3

The original paper mentions two corpora: CoNLL 2003 (apparently here now) and the "CMU Seminar Announcements Task". However according to the page linked in the question the actual NER was trained on a larger combination of corpora: Our big English NER models were trained on a mixture of CoNLL, MUC-6, MUC-7 and ACE named entity corpora, and as a result the ...

3

Don't have enough reputation to comment to a resource, so answering this myself. About Annoy Annoy is a library being used here for finding approximate nearest neighbours, approximate being the key word here. Understanding K-NN and Approximate NN Now, let's see what is the difference with example of a problem. Say you have 10 entities (words / ...

3

If all of your images are similar to this one(or have a small set of possible designs), you can simply reference the location (pixel-wise) on the image where this fields are and slice it. After slicing you can use any OCR algorithm to extract that data. If your data has more variation than that, you can use OCR on the entire image, which is usually a slow ...

3

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the forward() method of the BERT model, we see the following lines explaining the return types: outputs = (sequence_output, pooled_output,) + encoder_outputs[1:] # add hidden_states and attentions if they are here return outputs # sequence_output, ...

3

High resource languages are languages for which many data resources exist, making possible the development of machine-learning based systems for these languages. English is by far the most well resourced language. West-Europe languages are quite well covered, as well as Japanese and Chinese. Naturally low-resource languages are the opposite, that is ...

3

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text. BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other ...

Only top voted, non community-wiki answers of a minimum length are eligible