A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [nltk]

The Natural Language Toolkit is a Python library for computational linguistics.

Filter by
Sorted by
Tagged with
20
votes
4answers
21k views

How can I get a measure of the semantic similarity of words?

What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal: ...
15
votes
4answers
42k views

Similarity between two words

I'm looking for a Python library that helps me identify the similarity between two words or sentences. I will be doing Audio to Text conversion which will result in an English dictionary or non ...
8
votes
2answers
5k views

Is there an alternative to nltk in golang?

Golang is one of my favourite languages and I want to use it for a personal NLP/ML project. Is golang's ecosystem good and rich enough for this? Is there an alternative package for nltk in golang?
8
votes
1answer
4k views

Complex Chunking with NLTK

I am trying to figure out how to use NLTK's cascading chunker as per Chapter 7 of the NLTK book. Unfortunately, I'm running into a few issues when performing non-trivial chunking measures. Let's ...
7
votes
6answers
7k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...
5
votes
1answer
135 views

Inferring Relational Hierarchies of Words

I am new to natural language processing and I have not heard of a problem similar to mine yet. I was wondering if anyone could refer me to a method for solving my problem, or tell me how this problem ...
5
votes
1answer
33 views

Chunking Sentences with Spacy

I have a lot of sentences (500k) which looks like this: ...
4
votes
2answers
6k views

NLTK Sklearn Genism Text to Topic

I aint no data scientist/machine learner. What Im Lookin for ...
4
votes
1answer
1k views

Creating training data

My task is to classify free text originated from customer complaints about our product. I have created a Taxonomy and have around 10 different categories. I've realized that these categories include ...
4
votes
1answer
5k views

How to extract Question/s from document with NLTK?

How to extract Only Question/s from document with NLTK ? Can we categorise this Question into Y/N and details type answerable ? Note: I am one week old in NLTK ;-)
4
votes
1answer
782 views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
4
votes
1answer
953 views

Group similar words under one topic and assign them a title

I am working on a natural language processing data problem and I have selected some keywords from it as features. I want to group them under one heading. But I can't find any method or algorithm to do ...
3
votes
2answers
294 views

What is the tag mapping for entity recognition in nltk?

When doing entity recognition using NLTK, one gets as a result a Tree with a bunch of words mapped to tags (eg. Mark -> NNP, <...
3
votes
1answer
4k views

How to change plot size in nltk.plot()

I'm following along the NLTK book and would like to change the size of the axes in a lexical dispersion plot: ...
3
votes
4answers
2k views

Sentiment Analysis of Movie Reviews using Python

I am currently doing sentiment analysis using Python. Here I am taking all the reviews from movie dataset and using Naive Bayes algorithm to predict whether the review is positive or negative. From ...
3
votes
1answer
271 views

Document Categorization Problem

I'm very new to data science in general, and have been tasked with a big challenge. My organization has a lot of documents that are all sorted on document type (not binary format, but a subjectively ...
3
votes
2answers
3k views

StanfordTokenizer will be deprecated in version 3.2.5 Warning

I was testing the StanfordNERTagger using the NLTK wrapper and this warning appeared: ...
3
votes
2answers
1k views

Word analysis in Python

I have a list of documents which look like this: ["Display is flickering"] ["Battery charger is broken"] ["Hard disk is making noises"] These text documents are ...
3
votes
1answer
320 views

Python: validating the existence of NLTK data with database search

I need to pull the names of companies out of resumes. Thousands of them. I was thinking of using NLTK to create a list of possible companies, and then cross-referencing the list of strings with ...
3
votes
3answers
475 views

Machine learning or NLP approach to convert string about month ,year into dates

I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates. Example : last year last month into December 2018 string may ...
3
votes
2answers
592 views

Training NLP with multiple text input features

Question: How can I train a NLP model with discrete labels that is based on multiple text input features? Background: I'm trying to predict the difficulty of a 4-option multiple choice exam ...
3
votes
0answers
35 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
3
votes
0answers
91 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
3
votes
2answers
799 views

Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis

As a part of my university project, I am researching/developing a sentiment analysis model where in I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning ...
2
votes
3answers
496 views

Why do we have to remove most common words for text analysis?

I am trying to do sentiment analysis the task is to classify racist tweets from other tweets. And I have read many articles and many have mentioned to remove the most common 10 words from the column ...
2
votes
1answer
1k views

Why are Chunking and IOB tags necessary?

I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions: Why and how is Chunking helpful? Plus ...
2
votes
2answers
158 views

What can be done so that 'teacher' and 'teaches' are treated similar?

Who teaches English? Now, after tokenizing, stemming.. it gives me Who, teach, English In my list of word, I have a ...
2
votes
1answer
3k views

Unable to load NLTK in spark using PySpark

I have install NLTK and its working fine with the following code, I running in pyspark shell ...
2
votes
3answers
2k views

Text Classifier with multiple bag-of-words

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the ...
2
votes
3answers
57 views

TFIDF for very short sentences

I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's ...
2
votes
0answers
33 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
2
votes
0answers
173 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
2
votes
0answers
169 views

Data categorization

I'm using Google API to categorize and predict problems with laptops (Hardware, Software, Network, Customer satisfaction, No reason). I inserted my training data and categories are very well evaluated ...
1
vote
3answers
68 views

Detect if word is «common English» word or slang word

I have a huge list of short phrases, for example: ...
1
vote
2answers
234 views

NLP: To remove verb and find the match in a sentence [closed]

Is there a NLP method like stemming, lemmatisation to figure out the below? ...
1
vote
1answer
121 views

Is there a good German Stemmer?

What I tried: ...
1
vote
2answers
54 views

Doc2vec '-' symbol occurrence

Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'. Vector representations of these words are incorrectly classified by doc2vec. ['...
1
vote
1answer
111 views

Extracting Part of Speech (Source and Destinations) using text mining/NLP?

I need to extract the source and destination terms from the text documents using text mining / NLP / Information Retrieval ? Example input: I am travelling from New York to London. I am ...
1
vote
2answers
2k views

Extracting words belonging to a key from the text

I have a description of a product say mobile in this form - "5.5" (13.97cm) full hd 1920 x 1080 pixels ips screen gorilla glass display with 401ppi resolution - 450 nits brightness & contrast ...
1
vote
1answer
23 views

How do I use NLTK Text object with Re library?

I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error; ...
1
vote
1answer
44 views
1
vote
1answer
107 views

what is difference between set() and word_tokenize()?

...
1
vote
3answers
324 views

Classification: how to handle reviews/long english words in feature set with all other numerical features

I am currently working on an use case where feature set contains numeric values such as amount, as well as a review feature which contains long winded english text. the english text will very well ...
1
vote
1answer
420 views

Tokenize text with both American and English words

I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English. Consequently, I get 2 tokens for “organization” ...
1
vote
1answer
805 views

extract names in a list of names

I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing. I have a collection of names which does not have any specific pattern and ...
1
vote
1answer
270 views

Given one language ngram model, how do I compare likelihoods of two texts of different length?

Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model. How does one ...
1
vote
2answers
74 views

How to automatically find the sentiment?

I have written a program that scrapes data from the web and I have in possession about 5k sentences which I want to analyze. Part 1: I am just starting out in data science and wanted to know if ...
1
vote
1answer
626 views

NLP : Rules for chunking Verb Phrases

I have read a lot of documentation surrounding NP chunking but what about Verb Phrases? Has there been a fixed set of rules for ...
1
vote
1answer
17 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
1
vote
2answers
73 views

Difference between packaged sentiment analysis tools (TextBlob/NLTK) and training your own classifier?

I'm new to ML and training classifiers in practice, so I was just wondering what the difference was between the built-in sentiment tools of packages such as NLTK and TextBlob as compared to manually ...