Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [language-model]

The tag has no usage guidance.

0
votes
0answers
5 views

Hierachical Softmax Example In Deep Learning Book

I was learning neural network using the book "Deep Learning" by Ian Goodfellow, Yoshua Bengio and Aaron Courville. Section 12.4.3.2 has an example to show the computation cost improvement from ...
0
votes
0answers
6 views

dataset for human memory modelling and spaced repetition

I'm working with a trainable model of the human memory. The linked paper sets up a decay function to model how you slowly forget information between repetition sessions. Their spaced repetition ...
0
votes
0answers
13 views

generating text with neural network, how to judge against perplexity scores

To illustrate my problem, I'm using fastai's AWD-LSTM implementation. But the question is about judging perplexity scores in general. The code is included below, but only so you can reproduce my ...
0
votes
0answers
55 views

Language Models vs Word Embeddings

I wanted to clear up my understanding of both Language Models and Word Embeddings and how they are related if at all. Language Models: There is the older counting model which consists of Markov ...
0
votes
0answers
13 views

What is the correct way to calculate the entropy of a language model on a data-set of sentences?

I want to fit the parameters of my language model by minimizing the entropy/ maximizing likelihood of my language model on my data-set. However, I am uncertain as how I should go about doing this. ...
5
votes
2answers
158 views

Are there any good out-of-the-box language models for python?

I'm prototyping an application and I need a language model to compute perplexity on some generated sentences. Is there any trained language model in python I can readily use? Something simple like <...
4
votes
1answer
48 views

LSTM training/prediction with no starting sequence

ML newbie here. As an exercise, I'm trying to build a character based language model based on a simple 1 layer LSTM. Based on what I've learned about LSTMs, a common usage is to take in a sequence of ...
0
votes
0answers
23 views

What is in-domain and out-domain exactly in language model and How can we distinguish in-domain and out-of-domain data?

As mentioned in "Strategies for Training Large Scale Neural Network Language Models (Mikolov 2013)", to reduce the number of training tokens, we use from in-domain data. What is exactly in-domain in ...
0
votes
1answer
57 views

NLP - extract sentence parts related to people [closed]

Thank you for your help, I appreciate your time.
2
votes
1answer
255 views

Fasttext exception error

I'm trying to run language detection using Facebook's fastText through a Python script but I get this error when I load the model : Exception: fastText: Cannot load lid.176.bin due to C++ extension ...
4
votes
1answer
6k views

Word2Vec embeddings with TF-IDF

When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences. But there does not seem to be a way to specify weights for the words calculated for instance using ...
1
vote
1answer
373 views

Word2Vec, softmax function

I was going term by term through the softmax function for the word2vec (SKIP-GRAM) model. I found most definition of these functions to be not 'clear' so I modified the notation to make sure I ...
2
votes
1answer
96 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
1
vote
0answers
68 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and ...
3
votes
1answer
131 views

What tools are available for programming language parsing for ML?

I want to preform a machine learning task (e.g. supervised classification, clustering) on a corpus of programming language source code (lets say Python), and I'm looking for tools for parsing and ...
4
votes
3answers
1k views

Words to numbers faster lookup

I'm training an LSTM for sentiment analysis on a review dataset downloaded from here. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After ...
0
votes
1answer
118 views

Diminishing returns in language identification data set size?

Most problems have a curve whereby the results improve as data are added but level off at some point. Are there research papers or industry results that discuss the correlation between data set size ...
1
vote
1answer
175 views

Given one language ngram model, how do I compare likelihoods of two texts of different length?

Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model. How does one ...
17
votes
6answers
10k views

What is the difference between model hyperparameters and model parameters?

I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. ...
3
votes
2answers
464 views

how much text data is required for a meaningful use of word2vec

how much data does word2vec require? Are there any public data sets that are useful? For example, could it be that 1000 newspaper articles are enough to use word2vec? Here is a word2vec tutorial ...
6
votes
1answer
628 views

Can finite state machines be encoded as input/output for a neural network?

I want to encode finite state machines (specifically DFAs) as output (or input) of a neural network for a supervised learning task. Are there any ways in the literature for doing this? I've already ...
2
votes
0answers
140 views

In plain English, how to descibe i/o of the TensorFlow for language modelling?

I have followed the tutorial here about language modelling using Tensorflow to create LSTM and used PTB dataset. The code is here I failed to understnad the exact specific input and the output of the ...
3
votes
0answers
218 views

Stanford NER Training - Assign weight to each word

I am using Stanford NER to recognize each entity in a search text. Once I identify entities, I need to pass that entities to an algorithm which calculates score for each entity type (e.g. country, ...
3
votes
2answers
2k views

How does Alexa utterance parsing work?

What are the basic principles/tools necessary to make something like Alexa utterance parsing? For reference, Alexa allows a designer to define phrases with "placeholders" that will be filled in. For ...
1
vote
1answer
631 views

Neural Networks for Predictive typing

I don't have a background in neural networks. But, various studies has been proved that neural networks (feed forward / Recurrent) outperformed n-gram language modeling for predicting words in a ...
9
votes
5answers
7k views

How to create a good list of stopwords

I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and ...
4
votes
1answer
1k views

Improve CoreNLP POS tagger and NER tagger?

The CoreNLP parts of speech tagger and name entity recognition tagger are pretty good out of the box, but I'd like to improve the accuracy further so that the overall program runs better. To explain ...
8
votes
1answer
1k views

What is generative and discriminative model? How are they used in Natural Language Processing?

This question asks about generative vs. discriminative algorithm, but can someone give an example of the difference between these forms when applied to Natural Language Processing? How are generative ...