20

If you want to tackle the problem from another perspective, with an end to end learning, such that you don't specify ahead of time this large pipeline you've mentioned earlier, all you care about is the mapping between sentences and their corresponding SQL queries. Tutorials: How to talk to your database Papers: Seq2SQL: Generating Structured ...


20

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this: The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even a human translator will not necessarily score 1. It is important to note that ...


10

1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, numerical vector for each word signifies it's relation with its context words. These features are learnt by the neural network as this is unsupervised method. Each ...


10

I would say that lemmatization is generally the preferred way of reducing related words to a common base. This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from ...


7

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the ...


6

As you already understand, the vast majority of the data science work is made with rather high level languages such as Python and R. So it's not a matter of prevalence, it's a matter of which part in the big world of data science you want/can do with your skills and your tools. Imho inventing new models requires: strong theoretical background in maths and ...


6

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included. Paper with code contains a section on NLP. That's a great website ...


5

Yes. That should be fine. What you suggest makes sense and should not bias the results. The reasons you give are good ones. For any reasonable classifier, if the value of an attribute is always zero in the training set, that should cause the attribute to be essentially ignored. There is a simple test to let you confirm this. You can try, for each ...


5

NLTK has an excellent step by step guide on everything you need to convert human language to an SQL query using the nltk package in python. It’s rudimentary, but it answers your question.


5

The first line of the BERT abstract is We introduce a new language representation model called BERT. The key phrase here is "language representation model". The purpose of BERT and other natural language processing models like Word2Vec is to provide a vector representation of words, so that the vectors can be used as input to neural networks for other ...


5

BERT is a transformer. A transformer is made of several similar layers, stacked on top of each others. Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n. The hidden state you mention is simply the output of each layer. You might want to quickly look into this explanation of the Transformer architecture : ...


5

BLEU scores are based on comparing the translation to evaluate against a gold-standard translation. In general the gold-standard translation is the same source sentence translated by a professional translator, so in theory a professional human translation should always receive the maximum score of 1 (BLEU scores are normalized between 0 and 1). However it's ...


5

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...


5

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


4

Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$. Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}{P(w_1,\ldots,w_N)} = \frac{P(c) \cdot \prod_{i=1}^N P(w_i|c)}{P(w_1,\ldots,w_N)} \neq \prod_{i=1}^NP(c|w_i)$$ where the second equation holds because of the ...


4

You can use these tips : Should I exclude them for the corpus and from training the model? You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make a big difference in training. The model's generalisation power won't be compromised. should I manually translate them (Requesting natives from each ...


4

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not. The [SEP] in the middle is here to help the model understand which token belong to which sentence. At ...


4

Machine learning is inherently data intensive, and typical ML algorithms are massively data-parallel. Therefore, even when developing new algorithms, high-level mathy languages (like Python, R, Octave) can be reasonably fast if you are willing to describe your algorithm in terms of standard operations on matrices and vectors. On the other hand, for deeper ...


4

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...


4

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. ...


3

From the same blog, In this hybrid approach, we encode the subject and previous email by averaging the word embeddings in each field. We then join those averaged embeddings, and feed them to the target sequence RNN-LM at every decoding step. The BoW part of their hybrid approach is to get the general context of the email conversation by averaging the ...


3

For your first part of the question as to which question generation approaches are good - Neural question generation is being pretty popular (as of 2018/2019) among NLP enthusiasts but not all systems are great enough to be used directly in production. However, here are a few recent ones which reported the state-of-art performances in 2019 and have shared ...


3

Glove creates word vectors that capture meaning in vector space by taking global count statistics. The training objective of GloVe is to learn word vectors such that their dot product equals the logarithm of the words probability of co-occurrence. while optimizing this, you can use any number of hidden representations for word vector. In the original paper, ...


3

Yes, this is feasible. One-class classification is a thing, but it is usually used in a context where it is hard or impossible to get negative samples. In your case, I would argue, you can quite easily get tweets that are not about activism, therefore you can render it as a binary classification, because you have data points of two classes or labels: 1 for ...


3

I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows: criterion = nn.CrossEntropyLoss() total_loss = 0. ... for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)): ... loss = criterion(output.view(-1, ntokens), targets) loss.backward() total_loss +...


3

It is not uncommon that we can make sense of a sentence without reading it completely. Or when you are having a quick look at a document, you tend to oversee some words and still understand the main point. This is the intuition behind the word dropout. Generally this is done by randomly dropping each word in a sequence following for example a Bernoulli ...


3

I have used Siamese Bert and I can say it does a pretty good job. However, the issue is that the data that it has been fine-tuned atop of Bert may not necessarily, entirely represent the same semantic distance as with the answers between the true and the student's one. For instance, if there is a question about engineering, where a small change of word may ...


3

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used ...


2

The only way to obtain a high-quality dataset in your specific domain is to do it manually. There exists no other method that can give you the sentiment labels for texts in arbitrary domains. If there would exists such a method, why would you even bother to create your own model. You should probably find/hire people that can do this work for you. Adding ...


2

BIO(L) tagging is important (but as you correctly noted, not necessary) part of a NER pipeline. Main idea behind such split is to facilitate learning in following manner. Take English as an example, some words will (most likely) never end a Named Entity, like adjectives, so the model will never tag them as the L(ast) part of a named entity. The same applies ...


Only top voted, non community-wiki answers of a minimum length are eligible