Tag Info

Hot answers tagged ngrams

3

It's possible if you define CountVectorizer's token_pattern argument. If you're new to regular expressions, Python's documentation goes over how it deals with regular expressions using the re module (and scikit-learn uses this under the hood) and I recommend using an online regex tester like this one, which gives you immediate feedback on whether your ...

2

Well, as you state the problem, it is true that the search for a certain sequence of strings/words is the same as looking for the corresponding n-gram. However, keep in mind that an n-gram, when you use it for ML, (often) is represented as a factor. So a certain sequence of words or strings, is thought of carrying valuable information. Like „John Holmes“ ...

2

I will start with an advice - just google "n gram language model" and you will find a lot of good detailed explanations. With that being said I will give a short explanation about the "training phase" of n-gram language models (answer to question 2). The simplest way to build an N-gram language model strats with finding a big corpus - a ...

1

The traditional approach for this kind of problem would be an n-gram language model. The language model is trained on a large corpus, then it's reasonably simple to calculate the most likely missing tokens for any incomplete sentence. SRILM was one of the most common toolkits, but there are probably many other libraries.

1

A neural language model tries to predict a conditional probability $P (w_{i + 1} | w_1, \dots, w_i)$. It approximates the probability with the following $P(w_{i+1} | s(w_1, \dots, w_i))$, where $s$ is a state function. After that an LSTM looked at all the words $w_1, \dots, w_i$, it has an updated state, so now it contains some useful information about all ...

1

One was to evaluate the code is to run thousands of simulations and look the histogram of word length frequency. Then parameterize your code so long word bias can be changed up and down. Rerun the simulations and look to see if the histograms have changed.

1

On one side, for each compared pair of documents you would like the same word being represented the same way in both documents. That would be option 3. - 'global rare words'. On the other side, option 1. and 2. are more easily calculated and scaled with subsequent documents. I would try calculating rare words per document but skipping all n-grams with them ...

1

Tf-idf (text frequency - Inverness document frequency) will highlight your interests. https://en.m.wikipedia.org/wiki/Tf–idf

1

Without looking at the actual data, all we can really do is second guess and suggest best practices. Here are some pointers you can pursue - Gather more data - If it can be done, nothing like it. Improve data quality - The algorithm will always be as good as the data. Try extensive cleaning methods like - Lowercase (basic) so all your data is ...

1

TFIDF decreases as term frequency will be decreased linearly and idf increases log linearly. Document similarity will decrease as value of tfidf vectors should decrease as reputation of bigrams are more less than each single word.

1

Interesting question. I haven't worked with this sort of data much, but it seems to me that the bulk of the job is likely to be feature engineering. Every "supervised" statistical method that I know of requires that you shoe-horn your data into "outcomes" and "covariates." $\mathbf{Y}$ and $\mathbf{X}$. Once your data is in this form you simply find an ...

1

If your ultimate goal is to cluster similar categories and assuming that you have labels of each text as category1, category2,...,categoryN from 1 to N, a bag of words method would be sufficient in order to create features so that you can run multiple desired clustering algorithms. K-means can be a good starting point for getting similar groups of text ...

1

Recurrent Neural Network (RNN) create a single state vector over time. Thus that curve is to be expected. Initially, the state vector does have enough information to make a quality prediction. Then quickly reaches asymptotic performances. Overall, the predictions are between 15% and 22% correct. The shape of the graph might be a function of sentence length ...

Only top voted, non community-wiki answers of a minimum length are eligible