20
votes
Accepted
Similarity between two words
The closest would be like Jan has mentioned inhis answer, the Levenstein's distance (also popularly called the edit distance).
In information theory and computer science, the Levenshtein distance
...
- 8,256
19
votes
How can I get a measure of the semantic similarity of words?
Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg
The weather in California was _____ .
The ...
- 350
14
votes
Similarity between two words
Apart from very good responses here, you may try SequenceMatcher in difflib python library.
https://docs.python.org/2/library/difflib.html
...
- 281
10
votes
How can I get a measure of the semantic similarity of words?
In Text Analytic Tools for Semantic Similarity, they developed a algorithm in order to find the similarity between 2 sentences. But if you read closely, they find the similarity of the word in a ...
- 2,316
7
votes
Accepted
Is there a way to map words to their synonyms in tfidf?
TfIdf vectors require much more data than that to be useful, but also don't give you the ability to identify synonyms. To do that with vectors and the amount of data you're working with, you'll need a ...
- 650
6
votes
Similarity between two words
If your dictionary is not too big a common approach is to take the Levenshtein distance, which basically counts how many changes you have to make to get from one word to another. Changes include ...
- 9,348
6
votes
Is there a good German Stemmer?
Big problem and very good question!
I used spacy in the past, which has a German module. I guess stemming is not supported, but lemmatization.
Looking at the ...
- 7,366
5
votes
Accepted
Inferring Relational Hierarchies of Words
Look up taxonomy/ontology construction/induction. Relevant papers:
Automatic Taxonomy Construction from Keywords via Scalable Bayesian Rose Trees
Topic Models for Taxonomies
OntoLearn Reloaded. A ...
- 10.5k
5
votes
How to change plot size in nltk.plot()
Try something like this:
import matplotlib.pyplot as plt
plt.figure(figsize=(30, 20)) # the size you want
# your code goes here
- 4,296
5
votes
Accepted
Detect if word is «common English» word or slang word
It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain ...
- 5,456
4
votes
Similarity between two words
An old and well-known technique for comparison is the Soundex algorithm.
The idea is to compare not the words themselves but approximations of how they are pronounced. To what extent this actually ...
- 141
4
votes
Accepted
Document Categorization Problem
Except for the OCR part, the right bundle would be pandas and sklearn.
You can check this ipython notebook which uses ...
- 156
4
votes
Accepted
Creating training data
Assuming you are doing supervized learning to train a model that when deployed will take text as input and output a label (e.g., topic) or class probability, then what you probably want to do is ...
- 1,216
4
votes
NLTK Sklearn Genism Text to Topic
I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore.
You have basically two possibilities:
Classification of the text (...
- 41
4
votes
NLTK Sklearn Genism Text to Topic
To build off Mashimo's answer, one straightforward approach for topic modeling is "Latent Dirichlet Allocation" (LDA). The basic idea behind LDA is explained in this really good tutorial. Essentially, ...
- 341
4
votes
How to extract Question/s from document with NLTK?
Check out chapter 6 section 2.2 of the NLTK book.
EDIT: since apparently the community wants me to copy/paste stuff, here ya go:
2.2 Identifying Dialogue Act Types
When processing dialogue, it ...
- 3,208
4
votes
Where to know the list of NLTK tagset?
No one answered the question, then I will answer it myself. Thanks to @user12075 for the link. I didn't find it when I was googling it.
https://stackoverflow.com/questions/15388831/what-are-all-...
- 826
4
votes
Accepted
nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable"
There are a couple of items that could be improved in your code:
nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. ...
- 20.5k
4
votes
Is there a good German Stemmer?
The question is already almost 2 years old, but I guess many people struggle with the same question.
Many people use the TreeTagger for this. The TreeTagger does POS-Tagging and limmatization, but you ...
4
votes
Is there a good German Stemmer?
NLTK has an implementation of a stemmer specifically for German, called Cistem. I think it was added with NLTK version 3.4.
While the results on your examples look only marginally better, the ...
- 141
4
votes
Accepted
Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?
If you know there are only specific variants, you can obviously make a look-up table yourself (i.e. a Python dictionary).
Otherwise you could try using a fuzzy matching library, like fuzzywuzzy.
This ...
- 14.7k
4
votes
Is there an NLP corpus that contains common medical terms?
Welcome to the biomedical domain, one of the few domains in NLP where there are too many resources to choose from :)
Data resources:
Medline is a database corpus of 30 millions abstracts.
Each ...
- 25k
4
votes
Sentence tokenization for sentence without punctuation
You can apply a previous step to add punctuation and proper casing to the text, and then segment the sentences. For this, you may use Re-punctuate. When applied to your text, Re-punctuate gives the ...
- 23.8k
3
votes
Accepted
I need to measure Performance : AUC for this code of NLTK and skLearn
It is unclear if you are requesting AUC of ROC or Precision-Recall curve. However, instead of storing the indices of examples in sets, you can store the labels in lists and use sklearn's ...
- 1,173
3
votes
Filter phrases based on correspondent POS tags
Hey,
Here's how i would solve this.
The problem with regex is that you don't have any index so I find it better to convert your inputs into lists of single words/tags by splitting on spaces. Then it ...
- 31
3
votes
How can I get a measure of the semantic similarity of words?
Word2vec is a good starting point for most scenarios. It does capture semantics by way of prediction using CBOW method. It allows translations (as most repeated example I can put here again), V(King) -...
- 226
3
votes
Accepted
What is the tag mapping for entity recognition in nltk?
Tag mapping according to nltk source
...
- 161
3
votes
Accuracy of word and sent tokenize versus custom tokenizers in nltk
Why would we want a custom tokenizer?
Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer. Any toolkit needs to be flexible, and the ability to change ...
- 1,783
3
votes
Why are Chunking and IOB tags necessary?
BIO(L) tagging is important (but as you correctly noted, not necessary) part of a NER pipeline. Main idea behind such split is to facilitate learning in following manner.
Take English as an example, ...
- 779
3
votes
sentiment analysis nltk python
What you're doing right now is a traditional classification using supervised learning. This is a great method for predicting outcomes, but I suspect there are much better ways to complete this ...
Only top scored, non community-wiki answers of a minimum length are eligible