Questions tagged [language-model]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
36
votes
6answers
31k views

What is the difference between model hyperparameters and model parameters?

I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. ...
14
votes
4answers
5k views

Are there any good out-of-the-box language models for python?

I'm prototyping an application and I need a language model to compute perplexity on some generated sentences. Is there any trained language model in python I can readily use? Something simple like <...
11
votes
2answers
22k views

Word2Vec embeddings with TF-IDF

When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences. But there does not seem to be a way to specify weights for the words calculated for instance using ...
10
votes
1answer
1k views

What is generative and discriminative model? How are they used in Natural Language Processing?

This question asks about generative vs. discriminative algorithm, but can someone give an example of the difference between these forms when applied to Natural Language Processing? How are generative ...
9
votes
5answers
14k views

How to create a good list of stopwords

I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and ...
7
votes
2answers
4k views

What is purpose of the [CLS] token and why its encoding output is important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
6
votes
1answer
3k views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
5
votes
2answers
2k views

Can finite state machines be encoded as input/output for a neural network?

I want to encode finite state machines (specifically DFAs) as output (or input) of a neural network for a supervised learning task. Are there any ways in the literature for doing this? I've already ...
4
votes
1answer
410 views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
4
votes
3answers
4k views

Words to numbers faster lookup

I'm training an LSTM for sentiment analysis on a review dataset downloaded from here. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After ...
4
votes
1answer
185 views

How should I treat these non-English documents in the NLP task?

So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a ...
4
votes
1answer
1k views

Improve CoreNLP POS tagger and NER tagger?

The CoreNLP parts of speech tagger and name entity recognition tagger are pretty good out of the box, but I'd like to improve the accuracy further so that the overall program runs better. To explain ...
4
votes
1answer
152 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
3
votes
2answers
975 views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
3
votes
2answers
1k views

how much text data is required for a meaningful use of word2vec

how much data does word2vec require? Are there any public data sets that are useful? For example, could it be that 1000 newspaper articles are enough to use word2vec? Here is a word2vec tutorial ...
3
votes
1answer
201 views

What tools are available for programming language parsing for ML?

I want to preform a machine learning task (e.g. supervised classification, clustering) on a corpus of programming language source code (lets say Python), and I'm looking for tools for parsing and ...
3
votes
2answers
2k views

How does Alexa utterance parsing work?

What are the basic principles/tools necessary to make something like Alexa utterance parsing? For reference, Alexa allows a designer to define phrases with "placeholders" that will be filled in. For ...
3
votes
1answer
100 views

LSTM training/prediction with no starting sequence

ML newbie here. As an exercise, I'm trying to build a character based language model based on a simple 1 layer LSTM. Based on what I've learned about LSTMs, a common usage is to take in a sequence of ...
3
votes
1answer
83 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we're adding 1 to each word in our vocabulary but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me throw ...
3
votes
0answers
100 views

Fine tune gpt2 via huggingface API for domain specific LM

i am using the script in the examples folder to fine-tune the LM for a bot meant to deal with insurance related queries. So if someone were to type "i am looking to modify my ..." , the autocomplete ...
2
votes
1answer
315 views

What does 'Linear regularities among words' mean?

Context: In the paper "Efficient Estimation of Word Representations in Vector Space" by T. Mikolov et al., the authors make use of the phrase: 'Linear regularities among words'. What does that mean ...
2
votes
1answer
782 views

Neural Networks for Predictive typing

I don't have a background in neural networks. But, various studies has been proved that neural networks (feed forward / Recurrent) outperformed n-gram language modeling for predicting words in a ...
2
votes
2answers
49 views

Evaluating Language Model on specific topic

I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in ...
2
votes
1answer
868 views

Fasttext exception error

I'm trying to run language detection using Facebook's fastText through a Python script but I get this error when I load the model : Exception: fastText: Cannot load lid.176.bin due to C++ extension ...
2
votes
0answers
33 views

How to prepare the data for text generation task

First, I'm not sure whether the model contains the encoder during training. EOS means end-of-sentence. Encoder and decoder are part of transformer network. If without-encoder, training time: ...
2
votes
0answers
74 views

Build an Autocomplete model for document titles

I want to build an autocomplete model using RNN where input is article names (documents title). X: ['Billing', 'Loan status', 'Filling loan application', 'Contact Info', ...] The article name can ...
2
votes
0answers
152 views

In plain English, how to descibe i/o of the TensorFlow for language modelling?

I have followed the tutorial here about language modelling using Tensorflow to create LSTM and used PTB dataset. The code is here I failed to understnad the exact specific input and the output of the ...
2
votes
0answers
264 views

Stanford NER Training - Assign weight to each word

I am using Stanford NER to recognize each entity in a search text. Once I identify entities, I need to pass that entities to an algorithm which calculates score for each entity type (e.g. country, ...
1
vote
1answer
23 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
1answer
35 views

Comparing Language Model of two corpora

I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
1
vote
2answers
790 views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
1
vote
1answer
119 views

BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

For example I have an original sentence. The word barking corresponds to the word that is missing. ...
1
vote
1answer
49 views

Language modelling for Spell Checker

I am working on spell checkers, I want to create a spell checker, I am confused about which model to use Word-Level modelling Character-Level modelling plus I am preferring neural networks over ...
1
vote
1answer
110 views

Why Heaps' Law Equation looks so different in this NLP course?

I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed ...
1
vote
1answer
40 views

The principle of LM deep model

Language model(LM) is the task of predicting the next word. Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder. Or both with-...
1
vote
1answer
2k views

Word2Vec, softmax function

I was going term by term through the softmax function for the word2vec (SKIP-GRAM) model. I found most definition of these functions to be not 'clear' so I modified the notation to make sure I ...
1
vote
1answer
329 views

Given one language ngram model, how do I compare likelihoods of two texts of different length?

Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model. How does one ...
1
vote
0answers
44 views

For an n-Gram model with n>2, do we need more context at end of each sentence?

Jurafsky's book says we need to add context to left and right of a sentence: Does this mean, for example, if we've a corpus of three sentences: ...
1
vote
1answer
22 views

State-of-the-art Python packages that can evaluate language similarity

I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
1
vote
0answers
20 views

When using padding in sequence models, is Keras validation accuracy valid/ reliable?

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
1
vote
0answers
11 views

Effect of discounting parameter on Language Model Perplexity

The general formula for absolute discounting for calculating language model probabilities subtracts a discounting parameter d from the count of the ngram before calculating the probabilities. The ...
1
vote
1answer
25 views

Generate text using user-supplied keywords

I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need: User input: End-User: Data Scientists Region: Middle East ...
1
vote
0answers
11 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
1
vote
0answers
51 views

The differences between BNf and JSGF in NLP?

I wonder what the differences are between the BNF(Backus-Naur Form) and JSGF(Java Speech Grammar Format)? The former is a kind of context-free grammar taught in CS224, but I learned that the JSGF is ...
1
vote
1answer
41 views

Transfer learning between Language Model and classification

Following this fast.ai lecture, I am trying to understand the mechanism of Transfer Learning in NLP from a general Language Model (LM) to a classification problem. What is exactly taken from the ...
1
vote
0answers
16 views

Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode: ...
1
vote
0answers
29 views

Why is MLP working similar to RNN for text generation

I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus ...
1
vote
0answers
58 views

How is maximizing L(lambda1, lamda2, lamda3) equivalent to minimizing perplexity?

In language modeling, L(lambda1, lambda2, lambda3) is defined as: Sum(count of trigram(u,v,w) x q(w|u,v)) where u, v, w are words in the corpus and perplexity ...
1
vote
0answers
81 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and ...
0
votes
1answer
918 views

How to calculate perplexity in PyTorch?

I am wondering the calculation of perplexity of a language model which is based on ...