A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [nltk]

The Natural Language Toolkit is a Python library for computational linguistics.

Filter by
Sorted by
Tagged with
1
vote
0answers
212 views

NLP how to go beyond simple intent finding--using context and targeting objects

I'm manually applying NLP rules to a chatbot. Currently, I've a simple set of rules--actions that follow certain trigger words. Ex: "Create the match on saturday." This has been working for ...
0
votes
1answer
273 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
3
votes
0answers
35 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
3
votes
0answers
91 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
1
vote
0answers
160 views

TextRank using BM25F

I am trying to incorporate BM25F in textrank, I found out a scoring module http://whoosh.readthedocs.io/en/latest/api/scoring.html, but am unable to implement it. Has anyone incorporated this in ...
0
votes
1answer
301 views

Is there a synset for phrasal verbs?

Is there a database of phrasal verbs of similar semantics? Eg. one where querying ‘get in touch’ would return ‘get in contact with’, among other phrasal verbs of similar meanings? If not, given a ...
1
vote
2answers
234 views

NLP: To remove verb and find the match in a sentence [closed]

Is there a NLP method like stemming, lemmatisation to figure out the below? ...
1
vote
0answers
45 views

Predict words in a given corpus from jumbled incomplete characters [closed]

A complete newbie to data science. I have to predict the word using characters given. But the characters given can be in any order, and some might be missing or wrong. The problem is I get too many ...
0
votes
1answer
4k views

How to replace short words into full words from tweets using python

I am doing sentiment analysis on tweets. Most of the tweets contains short words and i want to replace them as original/full word. Suppose that tweet is: ...
2
votes
0answers
173 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
0
votes
2answers
66 views

How to select features for clustering to detect the number of different unique products in a search result?

I am trying to use clustering to determine the number of products in a search of products. So far I am using kmeans clustering. I have run into a problem where I cannot determine good features to use. ...
0
votes
1answer
86 views

How to read Feature Based Grammar from a string

To read Context Free Grammars we can use nltk.CFG.fromstring(""" S-> NP VP`... """) But it can't read a feature based grammar which is like? ...
0
votes
1answer
223 views

Proper/Possible methods for extracting unstructured data from websites

I'm working in Python, using Scrapy, and NLTK to try to understand how I can extract data from college websites. My scraper can navigate through the university websites and find their tuition fees ...
2
votes
1answer
1k views

Why are Chunking and IOB tags necessary?

I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions: Why and how is Chunking helpful? Plus ...
0
votes
1answer
41 views

Inappropriate stemming in nltk.stem

I was starting to learn stemming with nltk and a few words were quite inappropriately stemmed. For example:- very was stemmed to 'veri', important to 'import', once to 'onc', poorly to 'poorli' , etc....
4
votes
1answer
5k views

How to extract Question/s from document with NLTK?

How to extract Only Question/s from document with NLTK ? Can we categorise this Question into Y/N and details type answerable ? Note: I am one week old in NLTK ;-)
4
votes
1answer
782 views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
0
votes
1answer
1k views

Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a ...
3
votes
2answers
293 views

What is the tag mapping for entity recognition in nltk?

When doing entity recognition using NLTK, one gets as a result a Tree with a bunch of words mapped to tags (eg. Mark -> NNP, <...
3
votes
2answers
3k views

StanfordTokenizer will be deprecated in version 3.2.5 Warning

I was testing the StanfordNERTagger using the NLTK wrapper and this warning appeared: ...
0
votes
2answers
88 views

Search the Number of occurrences of the particular words in data using Pandas.

I wanna Search how many times "bad" and "good" words are repeated in the data frame and visualize this with histogram.
2
votes
3answers
2k views

Text Classifier with multiple bag-of-words

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the ...
0
votes
1answer
86 views

How to go about text mining for suggestions/Tips in reviews for restaurants? [closed]

For example for restaurants reviews usually have suggestions like "Go in the evenings", "order the so and so sauce with this dish" or even "TIP: ask for the blah blah blah" How can I detect such ...
0
votes
1answer
143 views

Extracting specific information from multiple unstructured website [closed]

I want to extract specific information e.g the names of all professor teaching subject xyz from multiple college faculty directory. Each website has different structure and the content is not uniform (...
4
votes
1answer
953 views

Group similar words under one topic and assign them a title

I am working on a natural language processing data problem and I have selected some keywords from it as features. I want to group them under one heading. But I can't find any method or algorithm to do ...
1
vote
1answer
420 views

Tokenize text with both American and English words

I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English. Consequently, I get 2 tokens for “organization” ...
3
votes
2answers
798 views

Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis

As a part of my university project, I am researching/developing a sentiment analysis model where in I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning ...
-1
votes
1answer
436 views

I want to make a text extractor for extracting particular type of text from unstructured text data.

I have a lot of data and manually extracted annotations for the text. I was looking for any advice to automate the annotation extraction up to a good level of accuracy. Any advice is welcome.
2
votes
2answers
158 views

What can be done so that 'teacher' and 'teaches' are treated similar?

Who teaches English? Now, after tokenizing, stemming.. it gives me Who, teach, English In my list of word, I have a ...
1
vote
1answer
111 views

Extracting Part of Speech (Source and Destinations) using text mining/NLP?

I need to extract the source and destination terms from the text documents using text mining / NLP / Information Retrieval ? Example input: I am travelling from New York to London. I am ...
7
votes
6answers
7k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...
1
vote
0answers
154 views

How to dowload Wikileaks Cable Leaks documents as text corpus?

I'd like to perform NLP analysis on Wikileaks US Diplomatic Cable Leaks documents (https://wikileaks.org/plusd/), preferably as Python's NLTK3 corpus od Mongo DB documents. I couldn't find any option ...
3
votes
1answer
4k views

How to change plot size in nltk.plot()

I'm following along the NLTK book and would like to change the size of the axes in a lexical dispersion plot: ...
1
vote
1answer
805 views

extract names in a list of names

I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing. I have a collection of names which does not have any specific pattern and ...
4
votes
2answers
6k views

NLTK Sklearn Genism Text to Topic

I aint no data scientist/machine learner. What Im Lookin for ...
1
vote
1answer
270 views

Given one language ngram model, how do I compare likelihoods of two texts of different length?

Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model. How does one ...
-2
votes
1answer
887 views

I need to measure Performance : AUC for this code of NLTK and skLearn [closed]

The code below measures precision and recall and F-measure (source). How can I measure AUC? ...
0
votes
1answer
3k views

Filter phrases based on correspondent POS tags

I have two lists: The first one includes sentences and the second one includes parts of speech (POS) tags. I want to use POS tags as a filter and find phrases according to a pattern such as 'JJ JJ NNS'...
1
vote
2answers
74 views

How to automatically find the sentiment?

I have written a program that scrapes data from the web and I have in possession about 5k sentences which I want to analyze. Part 1: I am just starting out in data science and wanted to know if ...
20
votes
4answers
21k views

How can I get a measure of the semantic similarity of words?

What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal: ...
15
votes
4answers
42k views

Similarity between two words

I'm looking for a Python library that helps me identify the similarity between two words or sentences. I will be doing Audio to Text conversion which will result in an English dictionary or non ...
8
votes
2answers
5k views

Is there an alternative to nltk in golang?

Golang is one of my favourite languages and I want to use it for a personal NLP/ML project. Is golang's ecosystem good and rich enough for this? Is there an alternative package for nltk in golang?
2
votes
1answer
3k views

Unable to load NLTK in spark using PySpark

I have install NLTK and its working fine with the following code, I running in pyspark shell ...
0
votes
2answers
914 views

How to train NLTK Sequence labeling algorithm for using custom labels/Train set?

I am trying to recognize and classify the entity types based on the IOB/Sequence labeling. I was able to use nltk.ne_chunk() which is already trained to recognize named entities using their train set....
3
votes
4answers
2k views

Sentiment Analysis of Movie Reviews using Python

I am currently doing sentiment analysis using Python. Here I am taking all the reviews from movie dataset and using Naive Bayes algorithm to predict whether the review is positive or negative. From ...
3
votes
2answers
1k views

Word analysis in Python

I have a list of documents which look like this: ["Display is flickering"] ["Battery charger is broken"] ["Hard disk is making noises"] These text documents are ...
1
vote
2answers
2k views

Extracting words belonging to a key from the text

I have a description of a product say mobile in this form - "5.5" (13.97cm) full hd 1920 x 1080 pixels ips screen gorilla glass display with 401ppi resolution - 450 nits brightness & contrast ...
3
votes
1answer
271 views

Document Categorization Problem

I'm very new to data science in general, and have been tasked with a big challenge. My organization has a lot of documents that are all sorted on document type (not binary format, but a subjectively ...
4
votes
1answer
1k views

Creating training data

My task is to classify free text originated from customer complaints about our product. I have created a Taxonomy and have around 10 different categories. I've realized that these categories include ...
5
votes
1answer
135 views

Inferring Relational Hierarchies of Words

I am new to natural language processing and I have not heard of a problem similar to mine yet. I was wondering if anyone could refer me to a method for solving my problem, or tell me how this problem ...