20

If you want to tackle the problem from another perspective, with an end to end learning, such that you don't specify ahead of time this large pipeline you've mentioned earlier, all you care about is the mapping between sentences and their corresponding SQL queries. Tutorials: How to talk to your database Papers: Seq2SQL: Generating Structured ...


20

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this: The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even a human translator will not necessarily score 1. It is important to note that ...


10

I would say that lemmatization is generally the preferred way of reducing related words to a common base. This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from ...


7

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the ...


6

As you already understand, the vast majority of the data science work is made with rather high level languages such as Python and R. So it's not a matter of prevalence, it's a matter of which part in the big world of data science you want/can do with your skills and your tools. Imho inventing new models requires: strong theoretical background in maths and ...


6

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included. Paper with code contains a section on NLP. That's a great website ...


5

Yes. That should be fine. What you suggest makes sense and should not bias the results. The reasons you give are good ones. For any reasonable classifier, if the value of an attribute is always zero in the training set, that should cause the attribute to be essentially ignored. There is a simple test to let you confirm this. You can try, for each ...


5

NLTK has an excellent step by step guide on everything you need to convert human language to an SQL query using the nltk package in python. It’s rudimentary, but it answers your question.


5

The first line of the BERT abstract is We introduce a new language representation model called BERT. The key phrase here is "language representation model". The purpose of BERT and other natural language processing models like Word2Vec is to provide a vector representation of words, so that the vectors can be used as input to neural networks for other ...


5

BERT is a transformer. A transformer is made of several similar layers, stacked on top of each others. Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n. The hidden state you mention is simply the output of each layer. You might want to quickly look into this explanation of the Transformer architecture : ...


5

BLEU scores are based on comparing the translation to evaluate against a gold-standard translation. In general the gold-standard translation is the same source sentence translated by a professional translator, so in theory a professional human translation should always receive the maximum score of 1 (BLEU scores are normalized between 0 and 1). However it's ...


4

Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$. Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}{P(w_1,\ldots,w_N)} = \frac{P(c) \cdot \prod_{i=1}^N P(w_i|c)}{P(w_1,\ldots,w_N)} \neq \prod_{i=1}^NP(c|w_i)$$ where the second equation holds because of the ...


4

You can use these tips : Should I exclude them for the corpus and from training the model? You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make a big difference in training. The model's generalisation power won't be compromised. should I manually translate them (Requesting natives from each ...


4

Machine learning is inherently data intensive, and typical ML algorithms are massively data-parallel. Therefore, even when developing new algorithms, high-level mathy languages (like Python, R, Octave) can be reasonably fast if you are willing to describe your algorithm in terms of standard operations on matrices and vectors. On the other hand, for deeper ...


4

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. ...


4

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...


3

From the same blog, In this hybrid approach, we encode the subject and previous email by averaging the word embeddings in each field. We then join those averaged embeddings, and feed them to the target sequence RNN-LM at every decoding step. The BoW part of their hybrid approach is to get the general context of the email conversation by averaging the ...


3

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not. The [SEP] in the middle is here to help the model understand which token belong to which sentence. At ...


3

Yes, this is feasible. One-class classification is a thing, but it is usually used in a context where it is hard or impossible to get negative samples. In your case, I would argue, you can quite easily get tweets that are not about activism, therefore you can render it as a binary classification, because you have data points of two classes or labels: 1 for ...


3

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...


3

I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows: criterion = nn.CrossEntropyLoss() total_loss = 0. ... for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)): ... loss = criterion(output.view(-1, ntokens), targets) loss.backward() total_loss +...


3

It is not uncommon that we can make sense of a sentence without reading it completely. Or when you are having a quick look at a document, you tend to oversee some words and still understand the main point. This is the intuition behind the word dropout. Generally this is done by randomly dropping each word in a sequence following for example a Bernoulli ...


3

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


3

I have used Siamese Bert and I can say it does a pretty good job. However, the issue is that the data that it has been fine-tuned atop of Bert may not necessarily, entirely represent the same semantic distance as with the answers between the true and the student's one. For instance, if there is a question about engineering, where a small change of word may ...


2

BIO(L) tagging is important (but as you correctly noted, not necessary) part of a NER pipeline. Main idea behind such split is to facilitate learning in following manner. Take English as an example, some words will (most likely) never end a Named Entity, like adjectives, so the model will never tag them as the L(ast) part of a named entity. The same applies ...


2

Starting with a small chunk of documents (~50) as a training set with a set of rules implements would be a good starting point. Train using a few algorithms and see if your training accuracy is coming out as acceptable or better than the human accuracy. Then slowly increase your training batch size and see the impact on the results. Once you believe that ...


2

That's a good question. The answer is yes and no. No because the input layer of the CBOW model expects a fixed number of words. So you'll either always input 3 words or 9 words. Yes because you can however set the sequence length as 9 words and provide just 3 words as context while the remaining 6 words can just be zero vectors. Remember in a CBOW each ...


2

This paper might be useful.... https://arxiv.org/abs/1801.06146


2

To complement Fadi's answer, the following are other useful papers on NL to SQL methods. The major difference of these methods is that they support queries that should be answered using more than one table (joining different tables), however the Salesforce paper (and their dataset) is focused on queries on one table at a time. Learning a Semantic Neural ...


2

You could go with different approaches: Lemme point few, You could just extract the keywords or tokenize out of the query using libraries like spacy or nltk, both support german languages. Then go for like page ranking approach, based on query optimization: Uber. You could go for Attention based seq2seq model, where you feed the inputs as questions and the ...


Only top voted, non community-wiki answers of a minimum length are eligible