Questions tagged [nltk]

NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Filter by
Sorted by
Tagged with
0
votes
1answer
6k views

nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable"

While trying to remove stopwords using the nltk package, the following error occurred: ...
0
votes
1answer
221 views

Extracting email id gives error

I am extracting email ids and storing them into a new column variable, but I am getting the issue: ...
1
vote
2answers
242 views

nltk.corpus for data science related words?

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found: ...
1
vote
1answer
722 views

From most frequent words how to extract technical skill words

I've scrape 30 job description web and stored them into a list called job_desc where each item is a job description. ...
1
vote
0answers
76 views

How can I find colors in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
3
votes
1answer
454 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
1
vote
2answers
416 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
0answers
157 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
1
vote
1answer
762 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
1
vote
2answers
63 views

Doc2vec '-' symbol occurrence

Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'. Vector representations of these words are incorrectly classified by doc2vec. ['...
-1
votes
1answer
565 views

What is "Interpolated Absolute Discounting" smoothing method

I'm asked to implement "Interpolated Absolute Discounting" for a bigram language model for a text. First, I don't know what is it exactly. I guess it is an interpolation between different ngrams (...
1
vote
0answers
847 views

How can I calculate perplexity for a bigram model?

I didn't find any function in nltk to calculate the perplexity. There are some codes I found: ...
0
votes
2answers
6k views

How to use bigrams for a text of sentences?

I have a text which has many sentences. How can I use nltk.ngrams to process it? This is my code: ...
6
votes
3answers
3k views

Training NLP with multiple text input features

Question: How can I train a NLP model with discrete labels that is based on multiple text input features? Background: I'm trying to predict the difficulty of a 4-option multiple choice exam ...
1
vote
0answers
103 views

Feature matrix for email classification:

This is in continuation of my earlier post. In my previous model, I used just two features which couldn't fare well and gave 71% prediction (accuracy) score. Now, I'm trying to consider another ...
5
votes
3answers
1k views

Machine learning or NLP approach to convert string about month ,year into dates

I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates. Example : last year last month into December 2018 string may ...
1
vote
1answer
250 views

what is difference between set() and word_tokenize()?

...
1
vote
0answers
835 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
25 views

Automating scoring of answers for a given question

There is a topic 'X' on which students are asked to write an English passage. Given a question, I have multiple solutions with their scorings out of 10. For a given answer by a new candidate, one ...
1
vote
3answers
745 views

Classification: how to handle reviews/long english words in feature set with all other numerical features

I am currently working on an use case where feature set contains numeric values such as amount, as well as a review feature which contains long winded english text. the english text will very well ...
1
vote
1answer
664 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
1
vote
1answer
186 views

Sentiment analysis for multiple entry in one text

I would like to do sentiment analysis on a set of financial news from the S&P 500 for given entities (organization names). However, each news (rows in my dataset) may have more than one entity and ...
0
votes
3answers
2k views

Python Code to find the number of hapax legomena in a Text or Words_List

In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, either in the written record of an entire language, in the works of an author, or in a single text. The term ...
0
votes
5answers
26k views

remove special character in a List or String

Input_String is Text_Corpus of Jane Austen Book output Should be : ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question'] But getting this Output : ['to', 'be,', 'or', 'not', 'to', '...
1
vote
1answer
463 views

sentiment analysis nltk python

I'd like to perform sentiment analysis on stock comment using scikit and nltk. I already have about 100 comments on different stocks like "this stock will rock" which I marked as positive (1) or "this ...
2
votes
3answers
2k views

Why do we have to remove most common words for text analysis?

I am trying to do sentiment analysis the task is to classify racist tweets from other tweets. And I have read many articles and many have mentioned to remove the most common 10 words from the column ...
2
votes
1answer
3k views

Where to know the list of NLTK tagset?

I need to know what the abbreviations mean, e.g 'VBD', 'NN', etc. I am learning text processing and now I want to get verbs in the sentences, so that I need to know which abbreviations represent any ...
0
votes
0answers
264 views

NLP how to go beyond simple intent finding--using context and targeting objects

I'm manually applying NLP rules to a chatbot. Currently, I've a simple set of rules--actions that follow certain trigger words. Ex: "Create the match on saturday." This has been working for ...
1
vote
2answers
657 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
1
vote
0answers
39 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
3
votes
0answers
106 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
0
votes
1answer
595 views

Is there a synset for phrasal verbs?

Is there a database of phrasal verbs of similar semantics? Eg. one where querying ‘get in touch’ would return ‘get in contact with’, among other phrasal verbs of similar meanings? If not, given a ...
1
vote
2answers
687 views

NLP: To remove verb and find the match in a sentence [closed]

Is there a NLP method like stemming, lemmatisation to figure out the below? ...
1
vote
0answers
61 views

Predict words in a given corpus from jumbled incomplete characters [closed]

A complete newbie to data science. I have to predict the word using characters given. But the characters given can be in any order, and some might be missing or wrong. The problem is I get too many ...
0
votes
1answer
6k views

How to replace short words into full words from tweets using python

I am doing sentiment analysis on tweets. Most of the tweets contains short words and i want to replace them as original/full word. Suppose that tweet is: ...
2
votes
0answers
211 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
0
votes
2answers
75 views

How to select features for clustering to detect the number of different unique products in a search result?

I am trying to use clustering to determine the number of products in a search of products. So far I am using kmeans clustering. I have run into a problem where I cannot determine good features to use. ...
0
votes
1answer
115 views

How to read Feature Based Grammar from a string

To read Context Free Grammars we can use nltk.CFG.fromstring(""" S-> NP VP`... """) But it can't read a feature based grammar which is like? ...
0
votes
1answer
238 views

Proper/Possible methods for extracting unstructured data from websites

I'm working in Python, using Scrapy, and NLTK to try to understand how I can extract data from college websites. My scraper can navigate through the university websites and find their tuition fees ...
2
votes
1answer
2k views

Why are Chunking and IOB tags necessary?

I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions: Why and how is Chunking helpful? Plus ...
0
votes
1answer
85 views

Inappropriate stemming in nltk.stem

I was starting to learn stemming with nltk and a few words were quite inappropriately stemmed. For example:- very was stemmed to 'veri', important to 'import', once to 'onc', poorly to 'poorli' , etc....
5
votes
1answer
8k views

How to extract Question/s from document with NLTK?

How to extract Only Question/s from document with NLTK ? Can we categorise this Question into Y/N and details type answerable ? Note: I am one week old in NLTK ;-)
5
votes
1answer
1k views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
1
vote
1answer
2k views

Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a ...
3
votes
2answers
376 views

What is the tag mapping for entity recognition in nltk?

When doing entity recognition using NLTK, one gets as a result a Tree with a bunch of words mapped to tags (eg. Mark -> NNP, <...
3
votes
2answers
4k views

StanfordTokenizer will be deprecated in version 3.2.5 Warning

I was testing the StanfordNERTagger using the NLTK wrapper and this warning appeared: ...
0
votes
2answers
96 views

Search the Number of occurrences of the particular words in data using Pandas.

I wanna Search how many times "bad" and "good" words are repeated in the data frame and visualize this with histogram.
2
votes
4answers
3k views

Text Classifier with multiple bag-of-words

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the ...
0
votes
1answer
86 views

How to go about text mining for suggestions/Tips in reviews for restaurants? [closed]

For example for restaurants reviews usually have suggestions like "Go in the evenings", "order the so and so sauce with this dish" or even "TIP: ask for the blah blah blah" How can I detect such ...
0
votes
1answer
153 views

Extracting specific information from multiple unstructured website [closed]

I want to extract specific information e.g the names of all professor teaching subject xyz from multiple college faculty directory. Each website has different structure and the content is not uniform (...