Questions tagged [ngrams]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
18 views

Does N-gram language model for text generation are more efficient than Neural Network language models?

I recently build an language model with N-gram model for text generation and for change I started exploring Neural Network for text generation. One thing I observed that the previous model results ...
user avatar
  • 11
1 vote
1 answer
10 views

How to keep only the top k-frequent ngrams in a text field with pandas?

How to keep only the top k-frequent ngrams in a text field with pandas? For example, I've a text column. For every row in it, I only want to keep those substrings that belong to the top k-frequent ...
user avatar
  • 359
1 vote
1 answer
10 views

Application of bag-of-ngrams in feature engineering of texts

I've got few questions about the application of bag-of-ngrams in feature engineering of texts: How to (or can we?) perform word2vec on bag-of-ngrams? As the feature space of bag of n-gram increases ...
user avatar
  • 359
3 votes
0 answers
34 views

Understanding Kneser-Ney Formula for implementation

I am trying to implement this formula in Python $$ \frac{\text{max}(c_{KN}(w^{i}_{i-n+1} - d), 0)}{c_{KN}(w^{i-1}_{i-n+1})} + \lambda(c_{KN}(w^{i-1}_{i-n+1})\mathbb{P}(c_{KN}(w_{i}|w^{i-1}_{i-n+2})$$ ...
user avatar
  • 237
1 vote
1 answer
18 views

N-Gram Smoothing

I am wondering if there is a good example out there that compares N-Gram with various smoothing techniques. I found this notebook that applies Laplace transform but that is about it. Any suggestions ...
user avatar
  • 237
0 votes
0 answers
4 views

Size Matrix features after applying 6 1D Kernels on one-hot encoded vectors

Suppose we are building the following model to build a neural network over one-hot encoded vectors of characters: For a given dataset, it’s not reasonable to read the whole text! So, we take some ...
user avatar
  • 179
2 votes
0 answers
22 views

Classifying short strings of text with additional context

I have a list of short strings each identifying a city. Misspellings are very common. The example below shows some of these short strings, along with the correct city they're supposed to match. ...
user avatar
  • 165
0 votes
2 answers
132 views

How do I get ngrams for all combinations of words in a sentence?

Lets say I have a sentence "I need multiple ngrams". If I create bigrams using Tf idf vectorizer it will create bigrams only using consecutive words. i.e. I will get "I need", &...
user avatar
0 votes
0 answers
21 views

The best way/instruments to use a communication protocol messages in hex form as input parameter for machine/deep learning (n-grams?)

I am trying to categorize server software versions based on server responses to various slightly different hex messages. To extract the ML input parameters from these hex messages I suppose to use the ...
user avatar
  • 1
2 votes
1 answer
37 views

Usage of KL divergence to improve BOW model

For a university project, I chose to do sentiment analysis on a Google Play store reviews dataset. I obtained decent results classifying the data using the bag of words (BOW) model and an ADALINE ...
user avatar
  • 23
2 votes
1 answer
66 views

NLP: find the best preposition for connecting parts of a sentence

My task is to connect 2-3 parts of the sentence into one whole using a preposition the first part is some kind of action. Ex. "take pictures" the second part is an object that can ...
user avatar
1 vote
1 answer
40 views

what is the training phase in N-gram model?

Following is my understanding of N gram model used in text prediction case : Given a sentence say, " I love my " (say N = 1 /bigram), using N gram and say 4 possible candidates ( country, ...
user avatar
0 votes
0 answers
94 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
user avatar
  • 51
1 vote
0 answers
37 views

Self Organising Map with variable length ordered sets of N-grams

I want to preface my question with the highlighted situation I have might not be applicable to kohonen self organising maps (SOM) due to a lack of understanding on my part so I do apologise if that is ...
user avatar
  • 111
1 vote
0 answers
56 views

For an n-Gram model with n>2, do we need more context at end of each sentence?

Jurafsky's book says we need to add context to left and right of a sentence: Does this mean, for example, if we've a corpus of three sentences: ...
user avatar
  • 113
3 votes
1 answer
200 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we are adding 1 to each word in our vocabulary, but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me ...
user avatar
  • 113
6 votes
2 answers
176 views

N-grams for RNNs

Given a word $w_{n}$ a statistical model such a Markov chain using n-grams predicts the subsequent word $w_{n+1}$. The prediction is by no means random. How is this translated into a neural model? I ...
user avatar
  • 61
3 votes
1 answer
235 views

FastText Model Explained

I was reading the FastText paper and I have a few questions about the model used for classification. Since I am not from NLP background, some I am unfamiliar with the jargon. In the figure, what ...
user avatar
0 votes
1 answer
39 views

Why is n-grams language independent?

I don't understand how n-grams are language independent. I've read that by using character n-grams of a word than the word itself as dimensions of a vector space model, we can skip the language-...
user avatar
  • 157
1 vote
1 answer
78 views

Is the search for a specific n-gram the same like a string search?

Is the result of a search for a specific n-gram like sherlock+holmes equal to the result of a regex search for "sherlock holmes" in the same document corpus? So if i read about n-grams for certain ...
user avatar
0 votes
1 answer
352 views

How to feed data for ngram model?

I want to train an ngram language model Let's say I have the following corpus: ...
user avatar
4 votes
1 answer
31 views

Artificially increasing frequency weight of word ending characters in word building

I have a database of letter pair bigrams. For example: ...
user avatar
  • 141
1 vote
0 answers
45 views

Dealing with missing n-grams in Naive Bayes classifier

I am doing sentiment analysis on code-mixed text data, i.e English used interchangeably with another language. The dataset I currently have is very small in size, approx 3.5k samples. I am sure that ...
user avatar
1 vote
0 answers
108 views

Skip-thought models applied to phrases instead of sentences

My goal is to build a statistical model with domain specific phrase embeddings. To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence ...
user avatar
0 votes
2 answers
227 views

Discarding rare words when comparing texts - per text, per comparison, or per codex?

I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency. first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words ...
user avatar
  • 584
2 votes
1 answer
2k views

How to customize word division in CountVectorizer?

...
user avatar
2 votes
1 answer
4k views

How to improve Naive Bayes?

I am solving a problem that address this question "What are the Actions that lead to high or low score?" I have the following Data that consist of text and score , I want to derive the words or ...
user avatar
  • 461
1 vote
1 answer
754 views

What affect will replacing words with bigrams have on TfIDF?

Say I have a corpus of text documents on which I have calculated each documents TfIDF vector. With this sparse matrix representation of the corpus, I can calculate similarities between documents by ...
user avatar
  • 1,494
1 vote
1 answer
36 views

Predicting interactions [closed]

First off, I don't really know much about machine learning. In a virtual world, such as a video game like minecraft or an application like Google Street view, a user can navigate the world using the ...
user avatar
  • 121
1 vote
0 answers
67 views

What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...
user avatar
  • 2,392
6 votes
2 answers
4k views

Clustering or classifing n-gram-based text categories

I have large set of data records looking like this: "text", "category" I extract n-grams from text (2-, 3- and 4-grams) and store count of each n-gram per ...
user avatar
  • 169
12 votes
1 answer
406 views

ngram and RNN prediction rate wrt word index

I tried to plot the rate of correct predictions (for the top 1 shortlist) with relation to the word's position in sentence : I was expecting to see a plateau sooner on the ngram setup since it ...
user avatar
  • 157