Hot answers tagged

3

Imho the most likely explanation is that the submission test set doesn't follow the same distribution as the training/validation/test data that you used to train and evaluate the model. In other words the test data that they use to evaluate is not a random sample from the full data, it's a different dataset collected independently, for example at a different ...


3

Without knowing your domain one cannot comment whether this is an appropriate size of feature names or not. However, consider this. Wordnet has database contains 155 327 words organized in 175 979 synsets for a total of 207 016 word-sense pairs[1]. Does your domain rely on more than 200% of the words in Wordnet. I'm familiar with sklearn's TF-IDF ...


2

If you want to be up to date with the new advancements, a good way is skimming through the accepted papers of the major NLP conferences, namely ACL, EMNLP, and the regional EACL, NAACL, AACL. If you want even more information, you can skim through the papers uploaded to the arxiv. One way to do that is via Twitter, by following bots that tweet papers in ...


2

If you have a fool-proof way of converting the Hindi transliteration to Hindi script (or vice versa), then you should do it, and you will have a monolingual corpus. If not, it is multilingual, from an algorithmic standpoint. Caveat: Even if we can convert all text into the same script reliably, we sometimes need to be careful about treating everything in the ...


2

Your problem as you said is a high level of syntax overlapping between your sentences. take a look at these two sentences: Work to live versus live to work. The earlier that you can allow yourself to enjoy other things in life, aside from your job while the latter means obtaining resources so that you can be a functional member of society, and to permit ...


2

Your understanding is correct: in the encoder-decoder attention blocks, the Keys and Values are the output of the encoder, while the Query vectors come from the decoder layers. At inference time we have as many Query positions as the step we are in. Remember that at inference time, the decoder behaves autoregressive, meaning that at each timestep T it ...


2

The distinction between supervised and unsupervised is a little bit tricky here. BERT pre-training is unsupervised with respect to the downstream tasks, but the pre-training itself is technically a supervised learning task. BERT is trained to predict words that have been masked in the input, so the target words are known at training time. The term ...


2

One can find sequence labelling libraries by searching for the term conditional random fields, the state of the art method. Probably one could also find libraries and tutorial by searching the term Named Entity Recognition, which is certainly the most standard NLP application of sequence labelling. Here are a few libraries that I know of: CRF++ crfsuite (...


1

we should obtain the vector of a document by averaging the vectors of all the words This is not necessarily the case. But surely it is a convenient approach. The main advantage in particular, is to avoid issues due to different lengths for different documents. By obtaining a single final vector, we make sure we can compare any document of any length. ...


1

This could be framed, as a first approximation, as a supervised learning classifier, where, based on the input texts (both name and description), you can build a series of features to build your classification model. One option is: tokenize (split into words) your texts (both name and descriptions) filter some not useful (presumably) words like ...


1

It's true that it's a bit of a complex process but it's worth understanding it in order to get the best out of the model. "Feature" and "attribute" (and probably observation but I'm not 100% sure) are the same thing. The features are the ones directly used by the model (as opposed to the raw input data). For every input word a vector of ...


1

You can either use a sentence embedding model to associate a vector to each of your inputs, and use a clustering algorithm like KMeans, or build a similarity matrix between your strings using a string distance metric, and use a similarity-based algorithm like Spectral Clustering or Agglomerative Clustering. The first one using KMeans might not work the best ...


1

The parameters in GPT-3, like any neural network, are the weights and biases of the layers. From the following table taken from the GTP-3 paper there are different versions of GPT-3 of various sizes. The more layers a version has the more parameters it has since it has more weights and biases. Regardless of the model version, the words it was trained on are ...


1

I think simple regex matching is all you need. Pass the tweet into a series of regular expressions that match emojis and hashtags and if nothing remains, discard.


1

Often it is not clear at beginning of a project how difficult a task is and which elements will have biggest impact. One approach is to setup a machine learning system to systematically evaluate options and empirically explore the problem. First setup the simplest possible text classification pipeline where the raw text enters the pipeline and "change&...


1

This looks like a sequence labelling problem, the most common such problem in NLP being Named Entity Recognition (NER). You'll find a lot of libraries and tutorials about NER. It can be done with Conditional Random Fields but there are also neural methods nowadays. Assuming your problem is not about standard entities (like persons names, organizations, ...


1

There is nothing wrong with an imbalanced training dataset. It's possible that no changes are required. When your training set is highly imbalanced like this, models in early training stages will predict everything to be the most prevalent class (positive in this case). After a longer training period, usually the model moves out of this local minima and ...


1

https://nlpprogress.com/ aims to provide pointers to the state of the art papers and datasets for the main NLP tasks. It seems to be updated regularly so far. However it depends on the efforts of volunteers so there's no guarantee about completeness or future updates.


1

Unfortunately, there is little theoretical knowledge about what complex neural networks do. Transformers are known to be universal approximations, so in theory they can learn to do any function with the input sentence, unlike the other alternatives that you mention. Most of the time, the accuracy of the BERT-like model would be strictly better. In practice, ...


1

Metrics for Q&A F1 score: Captures the precision and recall that words chosen as being part of the answer are actually part of the answer EM Score(exact match): which is the number of answers that are exactly correct (with the same start and end index). EM is 1 when characters of model prediction exactly matches True answers. The above scores are ...


1

EM (exact match) and F1 scores are typically calculated on different levels. EM is calculated on the character level. F1 is calculated on individual word level. Almost always, EM will be lower than F1. There is a good chance something is incorrect in the code. You should confirm your assumption by calculating the EM and F1 scores separately for empty answers ...


1

There is no genuinely unsupervised method for POS tagging; we can think of it as, Parts of speech are inferred by us, with rules defined by the specific language being tagged. There is no mathematical "notion" for a part of speech that we can conclude given some text without any predefined rule established empirically (Which is why it is not ...


1

Just to complement this thread with the latest AI updates on this matter: Open AI released recently Codex. Codex is an AI trained to convert and analyze many coding languages (main focus is python). It's also GPT-3 based.


1

Update for anyone googling this in 2021: Keras has implemented a MultiHead attention layer. If key, query, and value are the same, this is self-attention.


Only top voted, non community-wiki answers of a minimum length are eligible