Questions tagged [language-model]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
18 views

Language modelling for Spell Checker

I am working on spell checkers, I want to create a spell checker, I am confused about which model to use Word-Level modelling Character-Level modelling plus I am preferring neural networks over ...
0
votes
0answers
9 views

What's the best way to store BERT training data (input IDs)

The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
1
vote
0answers
10 views

Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode: ...
0
votes
1answer
19 views

How to feed data for ngram model?

I want to train an ngram language model Let's say I have the following corpus: ...
0
votes
0answers
279 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
0
votes
0answers
16 views

How to train a language model with bi lstm layers?

I am trying to understand how to train a LM using bi-LSTM in the case with "stack of bi LSTM". In the case of forward LSTM, we just need to add a classification layer on the top of the last hidden ...
3
votes
1answer
813 views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
0
votes
0answers
13 views

Make top N word predictions using a character model?

Given a character model that can predict (in addition to typical ascii characters) a special end-of-word character, I can make a word prediction by iteratively appending character predictions to my ...
4
votes
1answer
34 views

How should I treat these non-English documents in the NLP task?

So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a ...
0
votes
0answers
21 views

How to byte-pair encode character sequences at prediction time?

Suppose I have a vocabulary V = {'f', 'o', 'b', 'a', 'r', 'fo', 'ob', 'ar', 'foob', 'obar'} from BPE and at prediction time I come across an input ...
1
vote
1answer
46 views

Why Heaps' Law Equation looks so different in this NLP course?

I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed ...
1
vote
0answers
18 views

Why is MLP working similar to RNN for text generation

I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus ...
2
votes
0answers
31 views

How to prepare the data for text generation task

First, I'm not sure whether the model contains the encoder during training. EOS means end-of-sentence. Encoder and decoder are part of transformer network. If without-encoder, training time: ...
1
vote
1answer
35 views

The principle of LM deep model

Language model(LM) is the task of predicting the next word. Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder. Or both with-...
2
votes
1answer
158 views

What does 'Linear regularities among words' mean?

Context: In the paper "Efficient Estimation of Word Representations in Vector Space" by T. Mikolov et al., the authors make use of the phrase: 'Linear regularities among words'. What does that mean ...
0
votes
0answers
26 views

Long range dependency dataset

I am doing a project concerning formal languages and I want to relate them to DL structures. In this regard, I am looking for datasets that include enough long-range dependency for the use of ...
1
vote
0answers
26 views

How is maximizing L(lambda1, lamda2, lamda3) equivalent to minimizing perplexity?

In language modeling, L(lambda1, lambda2, lambda3) is defined as: Sum(count of trigram(u,v,w) x q(w|u,v)) where u, v, w are words in the corpus and perplexity ...
2
votes
0answers
66 views

Build an Autocomplete model for document titles

I want to build an autocomplete model using RNN where input is article names (documents title). X: ['Billing', 'Loan status', 'Filling loan application', 'Contact Info', ...] The article name can ...
11
votes
3answers
2k views

Are there any good out-of-the-box language models for python?

I'm prototyping an application and I need a language model to compute perplexity on some generated sentences. Is there any trained language model in python I can readily use? Something simple like <...
4
votes
1answer
78 views

LSTM training/prediction with no starting sequence

ML newbie here. As an exercise, I'm trying to build a character based language model based on a simple 1 layer LSTM. Based on what I've learned about LSTMs, a common usage is to take in a sequence of ...
0
votes
1answer
87 views

NLP - extract sentence parts related to people [closed]

Thank you for your help, I appreciate your time.
2
votes
1answer
609 views

Fasttext exception error

I'm trying to run language detection using Facebook's fastText through a Python script but I get this error when I load the model : Exception: fastText: Cannot load lid.176.bin due to C++ extension ...
7
votes
2answers
14k views

Word2Vec embeddings with TF-IDF

When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences. But there does not seem to be a way to specify weights for the words calculated for instance using ...
1
vote
1answer
913 views

Word2Vec, softmax function

I was going term by term through the softmax function for the word2vec (SKIP-GRAM) model. I found most definition of these functions to be not 'clear' so I modified the notation to make sure I ...
2
votes
1answer
134 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
1
vote
0answers
75 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and ...
3
votes
1answer
186 views

What tools are available for programming language parsing for ML?

I want to preform a machine learning task (e.g. supervised classification, clustering) on a corpus of programming language source code (lets say Python), and I'm looking for tools for parsing and ...
4
votes
3answers
3k views

Words to numbers faster lookup

I'm training an LSTM for sentiment analysis on a review dataset downloaded from here. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After ...
0
votes
1answer
137 views

Diminishing returns in language identification data set size?

Most problems have a curve whereby the results improve as data are added but level off at some point. Are there research papers or industry results that discuss the correlation between data set size ...
1
vote
1answer
259 views

Given one language ngram model, how do I compare likelihoods of two texts of different length?

Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model. How does one ...
23
votes
6answers
18k views

What is the difference between model hyperparameters and model parameters?

I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. ...
3
votes
2answers
755 views

how much text data is required for a meaningful use of word2vec

how much data does word2vec require? Are there any public data sets that are useful? For example, could it be that 1000 newspaper articles are enough to use word2vec? Here is a word2vec tutorial ...
5
votes
2answers
1k views

Can finite state machines be encoded as input/output for a neural network?

I want to encode finite state machines (specifically DFAs) as output (or input) of a neural network for a supervised learning task. Are there any ways in the literature for doing this? I've already ...
2
votes
0answers
146 views

In plain English, how to descibe i/o of the TensorFlow for language modelling?

I have followed the tutorial here about language modelling using Tensorflow to create LSTM and used PTB dataset. The code is here I failed to understnad the exact specific input and the output of the ...
2
votes
0answers
257 views

Stanford NER Training - Assign weight to each word

I am using Stanford NER to recognize each entity in a search text. Once I identify entities, I need to pass that entities to an algorithm which calculates score for each entity type (e.g. country, ...
3
votes
2answers
2k views

How does Alexa utterance parsing work?

What are the basic principles/tools necessary to make something like Alexa utterance parsing? For reference, Alexa allows a designer to define phrases with "placeholders" that will be filled in. For ...
1
vote
1answer
758 views

Neural Networks for Predictive typing

I don't have a background in neural networks. But, various studies has been proved that neural networks (feed forward / Recurrent) outperformed n-gram language modeling for predicting words in a ...
10
votes
5answers
10k views

How to create a good list of stopwords

I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and ...
4
votes
1answer
1k views

Improve CoreNLP POS tagger and NER tagger?

The CoreNLP parts of speech tagger and name entity recognition tagger are pretty good out of the box, but I'd like to improve the accuracy further so that the overall program runs better. To explain ...
8
votes
1answer
1k views

What is generative and discriminative model? How are they used in Natural Language Processing?

This question asks about generative vs. discriminative algorithm, but can someone give an example of the difference between these forms when applied to Natural Language Processing? How are generative ...