Questions tagged [language-model]

Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.

27 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
161 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
3
votes
1answer
112 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we're adding 1 to each word in our vocabulary but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me throw ...
3
votes
0answers
126 views

Fine tune gpt2 via huggingface API for domain specific LM

i am using the script in the examples folder to fine-tune the LM for a bot meant to deal with insurance related queries. So if someone were to type "i am looking to modify my ..." , the autocomplete ...
2
votes
0answers
38 views

How to prepare the data for text generation task

First, I'm not sure whether the model contains the encoder during training. EOS means end-of-sentence. Encoder and decoder are part of transformer network. If without-encoder, training time: ...
2
votes
0answers
74 views

Build an Autocomplete model for document titles

I want to build an autocomplete model using RNN where input is article names (documents title). X: ['Billing', 'Loan status', 'Filling loan application', 'Contact Info', ...] The article name can ...
2
votes
0answers
152 views

In plain English, how to descibe i/o of the TensorFlow for language modelling?

I have followed the tutorial here about language modelling using Tensorflow to create LSTM and used PTB dataset. The code is here I failed to understnad the exact specific input and the output of the ...
2
votes
0answers
267 views

Stanford NER Training - Assign weight to each word

I am using Stanford NER to recognize each entity in a search text. Once I identify entities, I need to pass that entities to an algorithm which calculates score for each entity type (e.g. country, ...
1
vote
0answers
47 views

For an n-Gram model with n>2, do we need more context at end of each sentence?

Jurafsky's book says we need to add context to left and right of a sentence: Does this mean, for example, if we've a corpus of three sentences: ...
1
vote
2answers
81 views

State-of-the-art Python packages that can evaluate language similarity

I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
1
vote
0answers
29 views

When using padding in sequence models, is Keras validation accuracy valid/ reliable?

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
1
vote
0answers
13 views

Effect of discounting parameter on Language Model Perplexity

The general formula for absolute discounting for calculating language model probabilities subtracts a discounting parameter d from the count of the ngram before calculating the probabilities. The ...
1
vote
0answers
13 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
1
vote
0answers
74 views

The differences between BNf and JSGF in NLP?

I wonder what the differences are between the BNF(Backus-Naur Form) and JSGF(Java Speech Grammar Format)? The former is a kind of context-free grammar taught in CS224, but I learned that the latter is ...
1
vote
1answer
63 views

Transfer learning between Language Model and classification

Following this fast.ai lecture, I am trying to understand the mechanism of Transfer Learning in NLP from a general Language Model (LM) to a classification problem. What is exactly taken from the ...
1
vote
0answers
20 views

Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode: ...
1
vote
0answers
41 views

Why is MLP working similar to RNN for text generation

I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus ...
1
vote
0answers
59 views

How is maximizing L(lambda1, lamda2, lamda3) equivalent to minimizing perplexity?

In language modeling, L(lambda1, lambda2, lambda3) is defined as: Sum(count of trigram(u,v,w) x q(w|u,v)) where u, v, w are words in the corpus and perplexity ...
1
vote
0answers
81 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and ...
0
votes
1answer
10 views

Training Objective of language model for GPT3

On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function: Our current objective weights every token equally and lacks a notion of what is most important to ...
0
votes
0answers
11 views

Language Model with Attention not learning

Language model with attention layer is not learning after 20 epochs. Both training and validation loss increase together, while the accuracy flattens at around 7% The way input data is pipelined is by ...
0
votes
0answers
7 views

Multilingual alternatives for med7

I'm looking for alternatives for med7 library for other common languages. Training a custom NER model for different languages seems like not the right option to ...
0
votes
0answers
13 views

Where does BERT fit in the Machine Learning Hierarchy?

I am a newbie in the machine learning world and I need guidance from the professionals. I am trying to make a hierarchy starting from machine learning, then to deep learning and to BERT. I have read ...
0
votes
0answers
20 views

Where does the evaluation speed advantage of Transformer-XL come from?

The Transformer-XL paper claims an advantage in evaluation speed 363x-1874x than that of a baseline Transformer model. However, I do not understand where this massive difference comes from. Although ...
0
votes
1answer
32 views

Optimal input setup for character-level text classification RNN

I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
0
votes
0answers
8 views

Help understanding input to biaxial network for generating music

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
0
votes
0answers
9 views

How to: Plot global mean precipitation data from the NDAA onto map regions?

Wondering how to/ if I can display amount of rainfall across regions (in env variable) on the map areas so that the points only pertain to separate languages and sized by rainsetPropn. Have used data ...
0
votes
0answers
15 views

Dirichlet smoothing as an IDF component

How can Dirichlet smoothing be used as an IDF component to estimate the probabilities of a Topic model ? i.e, Smoothing with a background collection model to estimate topic model ? I've seen many ...