# Tag Info

## Hot answers tagged natural-language-process

6

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included. Paper with code contains a section on NLP. That's a great website ...

5

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...

5

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...

3

I have used Siamese Bert and I can say it does a pretty good job. However, the issue is that the data that it has been fine-tuned atop of Bert may not necessarily, entirely represent the same semantic distance as with the answers between the true and the student's one. For instance, if there is a question about engineering, where a small change of word may ...

3

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used ...

2

Text is a 1D sequence, but is typically treated as a sequence of embedding vectors. So yes it is in some sense 2D input. But the embedding dimension doesn't really have any spatial meaning; adjacent dimensions aren't any more related than any others. There is no invariance across the embedding dimension either; the same values in one part of the embedding ...

2

So, how should I refine the word/sentence embeddings vector given by the BERT model in the case when I have a set completely unlabelled set of documents? What are you looking to achieve with these unlabelled documents? If you are looking to classify them, then there is no way of getting around getting labels and fine-tuning on them. I'm aware that ...

2

I am working in the same industry for a few years now and I can tell you that there are no publicly available datasets because of the nature of the documents. They are quite private and contains sensitive information, that are bound by rules and regulations.

2

Transformer based architectures are some of the most popular in NLP right now. You can check this blog post for more information: https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html Other than performance, one major advantage of transformers is that operations can be parallelized, making it much faster than RNNs/LSTMs.

2

As stated in 3.2 Model setup The generator G is made up of embedding layer, one bidirectional long short-term memory (BLSTM) [21] layer, one fully connected (FC) layer. And Gaussian noise is 10-dim vector concatenated with the output of BLSTM So the noise is concatenated to the embeddings computed by the BLSTM for each time step. I think they ...

2

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector. The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel&...

2

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that. However, you should take into account that it is possible that your model generates very similar sentences ...

2

First, congratulations for thinking to do a qualitative analysis of the results :) I know it should be obvious, but so many people just assume that the system works and don't bother checking their output. Now, strictly speaking what you're seeing is not a bug. These are errors made by a statistical system. A statistical system is not meant to get everything ...

2

There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components To solve your problem, I see at least three ways to do it. NTLK NLTK allows you to add known abbreviations as exceptions. See this StackOverflow post. Use a regular expression Since your problem is that you have some example of dots that shouldn't ...

2

Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence tagger and should behave better. Here is the issue with the details of its inception. You can train it if you have segmented sentence data. Another option is using NLTK's sent_tokenize, which should give better results than Spacy's ...

2

Generating artificial errors is generally risky in NLP, because it's difficult to make sure that the type and distribution of errors correspond exactly to real human errors. If the artificial errors diverge from real errors and a model is trained based on this data, the model will appear to have very good performance since it will rely on the patterns used ...

2

Technically this is sequence labeling, the most common application being Named Entity Recognition. However it looks like in this case you're trying to solve a problem of coreference resolution, which is a quite difficult task in general. I think this usually involves a more complex model than simple sequence labeling, but I'm not an expert in this. You might ...

1

I think there are (at least) two parts to take into account in evaluating such a model: Whether the generated text correctly relate to the input topic Whether the generated text is grammatically and semantically acceptable In my opinion the first kind of evaluation could reasonably be done with an automatic method such as the one you propose. Note that ...

1

As you certainly know, Machine Translation (MT) is a very challenging and useful task in the domain of Natural Language Processing (NLP). As such it is a very specialized research domain but also a very active area of research, and a very competitive one (in particular due to commercial applications, obviously). So there's a massive amount of research being ...

1

Well, in general case, machines do not understand the text, but they understand the numbers. Thus, we always tokenize the text followed by converting them to some form of numbers. We build a vocabulary of words from the given document, where each word can be assumed as a number corresponding to its index in the vocabulary. Further, this number is converted ...

1

They say they ignore the noise z in the input. Is that why they concat it later? So the over all model is more simple?

1

I'm not aware of any standard representation which increases the importance of document-frequent words, but IDF can simply be reverted: instead of the usual $$idf(w,D)=\log\left(\frac{N}{|d\in D\ |\ w \in d|}\right)$$ you could use the following: $$revidf(w,D)=\log\left(\frac{N}{|d\in D\ |\ w \notin d|}\right)$$ However for the task you describe I would be ...

1

@b_the_builder Nice finds! The first seems to me like an advancement of the Word Mower's distance by using the similarities between each word. I believe still may lack the domain adaptation. Whereas the second link you provided does the pre-training for that specific reason. All in all, whatever method you use I believe you will need to pick some ...

1

If you have a lot of data available to train, you should apply the techniques used in large transformer models, like GPT-2: very deep models (48 layers for the 1.5B parameters), modified initialization, pre-normalization, and reversible tokenization. You could also apply GPT-3's locally banded sparse attention patterns. If you have very small training data, ...

1

It's not mandatory. Removing stopwords can sometimes help and sometimes not. You should try both. A case for not using stopwords: Using stopwords will provide context to the user's intent. So when you use a contextual model like BERT, all stopwords are kept to provide enough context information like the negation words (not, nor, never) which are considered ...

1

In general stop-words can be omitted since they do not contain any useful information about the content of your sentence or document. The intuition behind that is that stop-words are the most common words in a language and occur in every document independent of the context. Therefore they contain no valuable information which could hint to the content of the ...

1

While ELMo was trained on English data, it does not know whether the data you give it as input is English or not. The input of ELMo is received at character-level. It may happen that the 1B Word data had hindi characters intermixed, case in which your characters would be encoded as they are or, most probably, your characters are encoded as unknown characters ...

1

So, your question concerns how to effectively translate the input words into their proposed correct words (e.g. Paruuuu --> Paru) via phonetics and spelling mistake corrections. My first idea on this would be to use a deep sequence to sequence model. In a sequence to sequence model, we encode the input word (e.g. Paruuuu) as a sequence of characters into ...

1

This is explained in the Methodology section of the paper: We calculate for each head how often it assigns its maximum attention weight (excluding EOS) to a token with which it is in one of the aforementioned dependency relations. We count each relation separately and allow the relation to hold in either direction between the two tokens. The nsubj ...

1

This corresponds to an NLP task called paraphrase detection. It's an active area of research, as far as I know there's no ready-to-use system able to perform this task very well, but there are probably a good few methods and prototypes around. A quick search gives these links for example: https://aclweb.org/aclwiki/Paraphrase_Identification_(...

Only top voted, non community-wiki answers of a minimum length are eligible