32

Hyperparameters and parameters are often used interchangeably but there is a difference between them. You call something a 'hyperparameter' if it cannot be learned within the estimator directly. However, 'parameters' is more general term. When you say 'passing the parameters to the model', it generally means a combination of hyperparameters along with some ...


26

In addition to the answer above. Model parameters are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: word frequency, sentence length, noun or verb distribution per sentence, the number of specific character n-grams per word, lexical diversity, etc. Model ...


10

Hyper-parameters are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while Parameters are those which would be learned by the machine like Weights and Biases.


9

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings: not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of them, and, ...


9

I also think that the first answer is incorrect for the reasons that @noob333 explained. But also Bert cannot be used out of the box as a language model. Bert gives you the p(word|context(both left and right) ) and what you want is to compute p(word|previous tokens(only left contex)). The author explains here why you cannot use it as a lm. However you can ...


9

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes an autoregressive factorization of the probability of the sentence: $p(s) = \prod_t P(w_t | w_{<t})$ On the other hand, BERT's masked LM loss focuses on ...


8

Let's say you are predicting the topic of a document given its words. A generative model describes how likely each topic is, and how likely words are given the topic. This is how it says documents are actually "generated" by the world -- a topic arises according to some distribution, words arise because of the topic, you have a document. Classifying ...


7

I think the accepted answer is incorrect. token.prob is the log-prob of the token being a particular type . I am guessing 'type' refers to something like POS-tag or type of named entity (it's not clear from spacy's documentation) and the score is a confidence measure over space of all types. This is not the same as the probabilities assigned by a language ...


6

In machine learning, a model $M$ with parameters and hyper-parameters looks like, $Y \approx M_{\mathcal{H}}(\Phi | D)$ where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is training data and $Y$ is output data (class labels in case of classification task). The objective during training is to find estimate of parameters $\hat{\Phi}$ ...


5

The spaCy package has many language models, including ones trained on Common Crawl. Language model has a specific meaning in Natural Language Processing (NlP). A language model is a probability distribution over sequences of tokens. Given a specific sequence of tokens, the model can assign a probability of that sequence appearing. SpaCy's language models ...


5

One approach would be to use tf-idf score. The words which occur in most of the queries will be of little help in differentiating the good search queries from bad ones. But ones which occur very frequently (high tf or term-frequency) in only few queries (high idf or inverse document frequency) as likely to be more important in distinguishing the good queries ...


5

By linear regularities among words, he meant that "Vectorized form of words should follow linear additive properties!" V("King") - V("Man") + V("Woman") ~ V("Queen)


5

phil ##am #mon is a subword encoding of the single word “philammon” into 3 tokens. The comment just means that they mask words as opposed to tokens by taking into account subword encoding. For more on subword encodings take a look at the slides from cs224, especially Byte Pair Encoding, from the Feb 14 subwords lecture at http://web.stanford.edu/class/...


4

In simplified words, Model Parameters are something that a model learns on its own. For example, 1) Weights or Coefficients of independent variables in Linear regression model. 2) Weights or Coefficients of independent variables SVM. 3) Split points in Decision Tree. Model hyper-parameters are used to optimize the model performance. For example, 1)...


4

A neural network is in principle a good choice when you have A LOT of similar data and classification tasks. Predicting the next character (or word... which is just multiple characters) is such a szenario. I don't think it really matters which kind of language you have, as long as you have enough training data of the same kind. See The Unreasonable ...


4

You can use these tips : Should I exclude them for the corpus and from training the model? You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make a big difference in training. The model's generalisation power won't be compromised. should I manually translate them (Requesting natives from each ...


4

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.


4

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. ...


3

Train a tfidfvectorizer with your corpus and use the following code: tfidf = Tfidfvectorizer () dict(zip(tfidf.get_feature_names(), tfidf.idf_))) Now you have a dictionary with words as its keys and weights as the corresponding values. Let me know if it worked.


3

I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows: criterion = nn.CrossEntropyLoss() total_loss = 0. ... for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)): ... loss = criterion(output.view(-1, ntokens), targets) loss.backward() total_loss +...


2

As word2vec is a neural network, it benefits from very large datasets. The Kaggle dataset is 50,000 reviews * ~5 sentences per review, so about a quarter million sentences. As they note, they get approximately the same results using bag of words and word2vec. One thing which is of note, since the review data comes from the internet, the sentences are much ...


2

You're doing something wrong. I can query a 100K word dict in nanoseconds word_list = open('/usr/share/dict/words').read().split() len(word_list) > 99171 word_dict = {word: hash(word) for word in word_list} %timeit word_dict['blazing'] > 10000000 loops, best of 3: 33.8 ns per loop


2

NLP stands for Natural Language Processing. Programming language source code are synthetic (or unnatural) languages. Thus, NLP tools are not useful for processing programming language source code. Understanding programming language source code is done by the compiler or interpreter. Compilers and interpreters perform many functions, including lexical ...


2

I'd like to extend the great @Emre's answer with another example - we are going to replace all tokenized words from the "1984" (c) George Orwell (120K words): In [163]: %paste import requests import nltk import pandas as pd # source: https://github.com/dwyl/english-words fn = r'D:\temp\.data\words.txt' url = 'http://gutenberg.net.au/ebooks01/0100021.txt' ...


2

You are probably trying to load word vectors that are shared in the fasttext.cc website or are trying to load using models that are trained with the latest one on master. There is an issue in github with a lot of comments. One solution is to try to load from the text files the code for which is given here. I was able to work around this using java but had to ...


2

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire sentence, they introduce a new tag. They can’t take any other word from the input sequence, because the output of that is the word representation. So they add a ...


2

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that. However, you should take into account that it is possible that your model generates very similar sentences ...


1

You might choose to demand predictions only after N steps of your sequence have elapsed. Then, predictions are trust-worthy. You've got to give your LSTM something to begin from, some context so to speak. Usually you sum the errors your network produced across all timesteps, but in such a case you ignore its outputs until the N'th timestep onwards. As ...


1

This is not a standard problem but you should be able to roughly do this using two basic kinds of tools that usually go together anyway: Use an NER system to identify people (as opposed to organizations) in sentences. Most systems have a default model that flags people. Use an Open Information Extraction system to get relation triples from sentences like (...


Only top voted, non community-wiki answers of a minimum length are eligible