Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [nltk]

The Natural Language Toolkit is a Python library for computational linguistics.

0
votes
1answer
10 views

Python Code to find the number of hapax legomena in a Text or Words_List

In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, either in the written record of an entire language, in the works of an author, or in a single text. The term ...
0
votes
4answers
26 views

remove special character in a List or String

Input_String is Text_Corpus of Jane Austen Book output Should be : ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question'] But getting this Output : ['to', 'be,', 'or', 'not', 'to', '...
0
votes
1answer
19 views

sentiment analysis nltk python

I'd like to perform sentiment analysis on stock comment using scikit and nltk. I already have about 100 comments on different stocks like "this stock will rock" which I marked as positive (1) or "this ...
2
votes
3answers
50 views

Why do we have to remove most common words for text analysis?

I am trying to do sentiment analysis the task is to classify racist tweets from other tweets. And I have read many articles and many have mentioned to remove the most common 10 words from the column ...
0
votes
1answer
24 views

Where to know the list of NLTK tagset?

I need to know what the abbreviations mean, e.g 'VBD', 'NN', etc. I am learning text processing and now I want to get verbs in the sentences, so that I need to know which abbreviations represent any ...
1
vote
0answers
36 views

NLP how to go beyond simple intent finding--using context and targeting objects

I'm manually applying NLP rules to a chatbot. Currently, I've a simple set of rules--actions that follow certain trigger words. Ex: "Create the match on saturday." This has been working for ...
0
votes
1answer
14 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
3
votes
0answers
13 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
3
votes
0answers
27 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
1
vote
0answers
20 views

TextRank using BM25F

I am trying to incorporate BM25F in textrank, I found out a scoring module http://whoosh.readthedocs.io/en/latest/api/scoring.html, but am unable to implement it. Has anyone incorporated this in ...
0
votes
1answer
47 views

Is there a synset for phrasal verbs?

Is there a database of phrasal verbs of similar semantics? Eg. one where querying ‘get in touch’ would return ‘get in contact with’, among other phrasal verbs of similar meanings? If not, given a ...
0
votes
0answers
17 views

Extract/detect IDs (like flight booking ids) from text

I'm looking to extract ids from the body of an email. The ids are similar to flight booking IDs. For example, in an email, I would like to obtain the booking reference (something like MNFF3RGC or MNF-...
0
votes
0answers
6 views

Nltk-posTag , merging into one list

I wanted to find number of nouns verbs and other stuffs in a paragraph using nltk. Since there are number of paragraphs, many lists are created under one name after I tag the paragraph. Now if I ...
1
vote
2answers
38 views

NLP: To remove verb and find the match in a sentence [closed]

Is there a NLP method like stemming, lemmatisation to figure out the below? ...
1
vote
0answers
16 views

Predict words in a given corpus from jumbled incomplete characters [closed]

A complete newbie to data science. I have to predict the word using characters given. But the characters given can be in any order, and some might be missing or wrong. The problem is I get too many ...
0
votes
1answer
414 views

How to replace short words into full words from tweets using python

I am doing sentiment analysis on tweets. Most of the tweets contains short words and i want to replace them as original/full word. Suppose that tweet is: ...
0
votes
0answers
27 views

Convert Lancaster stems back to one single word based on the stem?

I performed Stemming with Lancaster algorithm from the Python NLTK library, which is the most aggressive one, afaik. Here's an online tool that does this with small texts If I have words ...
1
vote
0answers
64 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
0
votes
0answers
391 views

How to extract a relation from a Named entity recognition model using NLTK in python

Using this sample article I have created a NLTK model which is able to perform named entity recognition - ...
0
votes
0answers
13 views

What is right approach for keyword extraction for a labelled data set

My Problem statement: I have a corpus of user comments, which are labelled with a sentiment of 1 or 0. I am trying to extra the keywords that would help me classify as 1 or 0. I am not interested in ...
0
votes
2answers
44 views

How to select features for clustering to detect the number of different unique products in a search result?

I am trying to use clustering to determine the number of products in a search of products. So far I am using kmeans clustering. I have run into a problem where I cannot determine good features to use. ...
0
votes
0answers
20 views

How to read Feature Based Grammar from a string

To read Context Free Grammars we can use nltk.CFG.fromstring(""" S-> NP VP`... """) But it can't read a feature based grammar which is like? ...
0
votes
1answer
103 views

Proper/Possible methods for extracting unstructured data from websites

I'm working in Python, using Scrapy, and NLTK to try to understand how I can extract data from college websites. My scraper can navigate through the university websites and find their tuition fees ...
1
vote
1answer
325 views

Why are Chunking and IOB tags necessary?

I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions: Why and how is Chunking helpful? Plus ...
0
votes
0answers
27 views

How to identify how survey takers answer a double barreled question?

I am analyzing the results of a survey where a survey taker is asked a, "double-barreled question." I am applying text analytics in order to answer the question, "which part of the question are survey ...
0
votes
1answer
26 views

Inappropriate stemming in nltk.stem

I was starting to learn stemming with nltk and a few words were quite inappropriately stemmed. For example:- very was stemmed to 'veri', important to 'import', once to 'onc', poorly to 'poorli' , etc....
3
votes
1answer
1k views

How to extract Question/s from document with NLTK?

How to extract Only Question/s from document with NLTK ? Can we categorise this Question into Y/N and details type answerable ? Note: I am one week old in NLTK ;-)
4
votes
1answer
501 views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
0
votes
1answer
767 views

Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a ...
3
votes
2answers
100 views

What is the tag mapping for entity recognition in nltk?

When doing entity recognition using NLTK, one gets as a result a Tree with a bunch of words mapped to tags (eg. Mark -> NNP, <...
3
votes
2answers
2k views

StanfordTokenizer will be deprecated in version 3.2.5 Warning

I was testing the StanfordNERTagger using the NLTK wrapper and this warning appeared: ...
0
votes
2answers
59 views

Search the Number of occurrences of the particular words in data using Pandas.

I wanna Search how many times "bad" and "good" words are repeated in the data frame and visualize this with histogram.
2
votes
3answers
833 views

Text Classifier with multiple bag-of-words

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the ...
0
votes
0answers
312 views

Extracting place/organization name from card transaction SMS

From the credit/debit card transaction SMS (example below), I want to extract the place/organization name where the money was spent. Sample SMS Thank you for using your SBI Debit Card 607XX8310 ...
0
votes
1answer
61 views

How to go about text mining for suggestions/Tips in reviews for restaurants? [closed]

For example for restaurants reviews usually have suggestions like "Go in the evenings", "order the so and so sauce with this dish" or even "TIP: ask for the blah blah blah" How can I detect such ...
0
votes
1answer
88 views

Extracting specific information from multiple unstructured website [closed]

I want to extract specific information e.g the names of all professor teaching subject xyz from multiple college faculty directory. Each website has different structure and the content is not uniform (...
3
votes
1answer
413 views

Group similar words under one topic and assign them a title

I am working on a natural language processing data problem and I have selected some keywords from it as features. I want to group them under one heading. But I can't find any method or algorithm to do ...
0
votes
1answer
164 views

Tokenize text with both American and English words

I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English. Consequently, I get 2 tokens for “organization” ...
2
votes
0answers
387 views

Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis

As a part of my university project, I am researching/developing a sentiment analysis model where in I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning ...
-1
votes
1answer
274 views

I want to make a text extractor for extracting particular type of text from unstructured text data.

I have a lot of data and manually extracted annotations for the text. I was looking for any advice to automate the annotation extraction up to a good level of accuracy. Any advice is welcome.
2
votes
2answers
98 views

What can be done so that 'teacher' and 'teaches' are treated similar?

Who teaches English? Now, after tokenizing, stemming.. it gives me Who, teach, English In my list of word, I have a ...
1
vote
1answer
81 views

Extracting Part of Speech (Source and Destinations) using text mining/NLP?

I need to extract the source and destination terms from the text documents using text mining / NLP / Information Retrieval ? Example input: I am travelling from New York to London. I am ...
4
votes
3answers
3k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...
1
vote
0answers
104 views

How to dowload Wikileaks Cable Leaks documents as text corpus?

I'd like to perform NLP analysis on Wikileaks US Diplomatic Cable Leaks documents (https://wikileaks.org/plusd/), preferably as Python's NLTK3 corpus od Mongo DB documents. I couldn't find any option ...
3
votes
1answer
2k views

How to change plot size in nltk.plot()

I'm following along the NLTK book and would like to change the size of the axes in a lexical dispersion plot: ...
0
votes
0answers
70 views

Efficient way of extracting full/complete sentence from tweet

I'm struggling to extract correct syntactically sentence from a tweet, ultimate goal is to build a paragraph from tweets. All what I found is the tokenizing and parsing but my goal is not to ...
1
vote
1answer
551 views

extract names in a list of names

I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing. I have a collection of names which does not have any specific pattern and ...
4
votes
2answers
5k views

NLTK Sklearn Genism Text to Topic

I aint no data scientist/machine learner. What Im Lookin for ...
1
vote
1answer
159 views

Given one language ngram model, how do I compare likelihoods of two texts of different length?

Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model. How does one ...
-2
votes
1answer
637 views

I need to measure Performance : AUC for this code of NLTK and skLearn [closed]

The code below measures precision and recall and F-measure (source). How can I measure AUC? ...