Questions tagged [ngrams]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
15 views

How to feed data for ngram model?

I want to train an ngram language model Let's say I have the following corpus: ...
0
votes
0answers
7 views

How to vectorize unigrams character to use LSH functions?

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate vector ...
0
votes
0answers
11 views

How to set 160 bit vector and assign corresponding ngram

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate ...
1
vote
0answers
13 views

Artificially increasing frequency weight of word ending characters in word building

I have a database of letter pair bigrams. For example: ...
1
vote
0answers
22 views

Dealing with missing n-grams in Naive Bayes classifier

I am doing sentiment analysis on code-mixed text data, i.e English used interchangeably with another language. The dataset I currently have is very small in size, approx 3.5k samples. I am sure that ...
0
votes
0answers
104 views

Anomaly detection in text - how to utilize ngram frequency of words for the detection of an anomalous document?

I am calculating ngram frequency on a text, and for each word I output its conditional probability: how frequent it is given thew last n-1 words, or, if you will, p(ngram)/p([n-1]gram). Using this I ...
1
vote
0answers
57 views

Skip-thought models applied to phrases instead of sentences

My goal is to build a statistical model with domain specific phrase embeddings. To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence ...
0
votes
1answer
279 views

Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?

The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
0
votes
2answers
80 views

Discarding rare words when comparing texts - per text, per comparison, or per codex?

I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency. first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words ...
0
votes
1answer
724 views

How to customize word division in CountVectorizer?

...
2
votes
1answer
1k views

How to improve Naive Bayes?

I am solving a problem that address this question "What are the Actions that lead to high or low score?" I have the following Data that consist of text and score , I want to derive the words or ...
1
vote
1answer
484 views

What affect will replacing words with bigrams have on TfIDF?

Say I have a corpus of text documents on which I have calculated each documents TfIDF vector. With this sparse matrix representation of the corpus, I can calculate similarities between documents by ...
1
vote
1answer
34 views

Predicting interactions [closed]

First off, I don't really know much about machine learning. In a virtual world, such as a video game like minecraft or an application like Google Street view, a user can navigate the world using the ...
1
vote
0answers
35 views

What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...