# Tag Info

20

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this: The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even a human translator will not necessarily score 1. It is important to note that ...

10

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT. They also have a github repo which is easy to work with.

8

Other options would be to... Compare similar text sequences, Compare similar string sequences, Use fuzzy matching. Fuzzy Matching: library(fuzzyjoin) # https://stackoverflow.com/questions/26405895/how-can-i-match-fuzzy-match-strings-from-two-datasets a <- data.frame(name = c('Ace Co', 'Bayes', 'asd', 'Bcy', 'Baes', 'Bays'), price = c(10,...

7

Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two? There is a confusion here: NLP is the whole domain of AI which deals with natural language. It includes virtually any task related to processing language data (usually mostly written data, but that's not the point). Topic modeling is ...

7

The reason you're seeing BERT and its derivatives as benchmarks is probably because it is newer than the other models mentioned and shows state-of-the-art performance on many NLP tasks. Thus, when researchers publish new models they normally want to compare them to the current leading models out there (i.e BERT). I don't know if there has been a study on the ...

7

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes an autoregressive factorization of the probability of the sentence: $p(s) = \prod_t P(w_t | w_{<t})$ On the other hand, BERT's masked LM loss focuses on ...

6

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included. Paper with code contains a section on NLP. That's a great website ...

5

It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that ...

5

I will take as reference fairseq's implementation of the Transformer model. With this assumption: In the transformer, masks are used for two purposes: Padding: in the multi-head attention, the padding tokens are explicitly ignored by masking them. This corresponds to parameter key_padding_mask. Self-attention causality: in the multi-head attention blocks ...

5

BLEU scores are based on comparing the translation to evaluate against a gold-standard translation. In general the gold-standard translation is the same source sentence translated by a professional translator, so in theory a professional human translation should always receive the maximum score of 1 (BLEU scores are normalized between 0 and 1). However it's ...

4

2 approaches to correct misspellings: Make your own dictionary of corrections, for example: mispell_dict = {'colour': 'color', 'centre': 'center', 'favourite': 'favorite', 'travelling': 'traveling', 'counselling': 'counseling', 'theatre': 'theater', 'cancelled': 'canceled', 'labour': 'labor', 'organisation': 'organization', 'wwii': 'world war 2', 'citicise'...

4

BERT generates contextualized word embeddings, which means that BERTprovides the most accurate embeddings when a word is in a sentence(context). For each of the words within the sentence, BERT will generate a vector of numbers. In your case, you will have a good representation of the word "bank". So if you have a sentence for all the other words that you ...

4

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. ...

4

A way to speed up this process is to preprocess the large dataset, the goal being to store the documents from A in a way which avoids a lot of useless comparisons. Store each document from A in an inverted index $m$, so that for any word $w$ $m[w]$ is the list of all documents in A which contain word $w$ (note that a document can appear several times in ...

4

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it. The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-...

4

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...

4

You need character embeddings. I assume you are already familiar with word2vec technology. Its goal it to make a model "learn" the relative meaning of words, placing them into a highly dimensional space. The same can be done with single characters, instead of whole words. The preprocessing steps you need will be a little bit different, but the ...

3

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not. The [SEP] in the middle is here to help the model understand which token belong to which sentence. At ...

3

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...

3

If all of your images are similar to this one(or have a small set of possible designs), you can simply reference the location (pixel-wise) on the image where this fields are and slice it. After slicing you can use any OCR algorithm to extract that data. If your data has more variation than that, you can use OCR on the entire image, which is usually a slow ...

3

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the forward() method of the BERT model, we see the following lines explaining the return types: outputs = (sequence_output, pooled_output,) + encoder_outputs[1:] # add hidden_states and attentions if they are here return outputs # sequence_output, ...

3

High resource languages are languages for which many data resources exist, making possible the development of machine-learning based systems for these languages. English is by far the most well resourced language. West-Europe languages are quite well covered, as well as Japanese and Chinese. Naturally low-resource languages are the opposite, that is ...

3

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text. BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other ...

3

Some common approaches to this problem are: Keep only the n- most common words in a corpus (automatically done in scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer). Keep all words, but downweight uninformative words using a transformation ...

3

Yes, this is feasible. One-class classification is a thing, but it is usually used in a context where it is hard or impossible to get negative samples. In your case, I would argue, you can quite easily get tweets that are not about activism, therefore you can render it as a binary classification, because you have data points of two classes or labels: 1 for ...

3

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...

3

Text summarization can be divided into two categories 1. Extractive Summarization and 2. Abstractive Summarization Extractive Summarization: These methods rely on extracting several parts, such as phrases and sentences, from a piece of text and stack them together to create a summary. Therefore, identifying the right sentences for summarization is of ...

3

I don't think there's anything close to doing this: It would be very hard to even define the task objectively, as different humans wouldn't agree about what is credible or not. It would require a complex system to represent reliable background knowledge... and again people wouldn't agree what should be considered "reliable" or not. Generally the state of ...

3

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.

Only top voted, non community-wiki answers of a minimum length are eligible