Questions tagged [nltk]

The Natural Language Toolkit is a Python library for computational linguistics.

42 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
2answers
995 views

Training NLP with multiple text input features

Question: How can I train a NLP model with discrete labels that is based on multiple text input features? Background: I'm trying to predict the difficulty of a 4-option multiple choice exam ...
3
votes
0answers
100 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
2
votes
1answer
20 views

Weighting of words in lexicon based sentiment analysis

I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following: $$ Score = \frac{\sum_{i}{word_i}...
2
votes
1answer
284 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
2
votes
0answers
40 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
2
votes
1answer
246 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
2
votes
0answers
188 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
2
votes
0answers
172 views

Data categorization

I'm using Google API to categorize and predict problems with laptops (Hardware, Software, Network, Customer satisfaction, No reason). I inserted my training data and categories are very well evaluated ...
1
vote
0answers
21 views

How to group news by topics when they do not have labels

I have a dataset that includes 20000 news titles and 9 columns that are not relevant (boolean values) for labelling them. I have tried to extract top words using tokenizetion and tf-idf for n-grams. ...
1
vote
1answer
27 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
1
vote
0answers
22 views

Semi-Supervised Learning using NLP

I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also ...
1
vote
0answers
156 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
1
vote
0answers
19 views

Determining topic of text

I was wondering what I should be looking into if I want to measure the similarity between a paragraph and a corpus of text. For example, given a paragraph of text and the entire corpus of Data ...
1
vote
1answer
275 views

Elbow method for cosine distance

I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
1
vote
2answers
126 views

nltk.corpus for data science related words?

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found: ...
1
vote
0answers
42 views

How can I find colors in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
1
vote
0answers
140 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
1
vote
0answers
417 views

How can I calculate perplexity for a bigram model?

I didn't find any function in nltk to calculate the perplexity. There are some codes I found: ...
1
vote
0answers
102 views

Feature matrix for email classification:

This is in continuation of my earlier post. In my previous model, I used just two features which couldn't fare well and gave 71% prediction (accuracy) score. Now, I'm trying to consider another ...
1
vote
0answers
559 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
25 views

Automating scoring of answers for a given question

There is a topic 'X' on which students are asked to write an English passage. Given a question, I have multiple solutions with their scorings out of 10. For a given answer by a new candidate, one ...
1
vote
0answers
228 views

NLP how to go beyond simple intent finding--using context and targeting objects

I'm manually applying NLP rules to a chatbot. Currently, I've a simple set of rules--actions that follow certain trigger words. Ex: "Create the match on saturday." This has been working for ...
1
vote
0answers
38 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
1
vote
0answers
181 views

TextRank using BM25F

I am trying to incorporate BM25F in textrank, I found out a scoring module http://whoosh.readthedocs.io/en/latest/api/scoring.html, but am unable to implement it. Has anyone incorporated this in ...
1
vote
1answer
1k views

Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a ...
1
vote
0answers
163 views

How to dowload Wikileaks Cable Leaks documents as text corpus?

I'd like to perform NLP analysis on Wikileaks US Diplomatic Cable Leaks documents (https://wikileaks.org/plusd/), preferably as Python's NLTK3 corpus od Mongo DB documents. I couldn't find any option ...
1
vote
0answers
238 views

NLP : What are some common verbs surrounding organization names in text

I am trying to come up with some rules to detect named entities, specifically company or organization names in text. I think it makes sense to focus on verbs. There are a lot of ...
0
votes
0answers
14 views

Is there a way to find difference between topics in two languages using nlp

I want to analyse queries and their differences between two different languages English and Spanish in this case. I'm aware about topic modelling. I'm in search of any corpus available or any ...
0
votes
0answers
20 views

English Word popularity scoring

I'm new to data science and looking for the word popularity ranking algorithm. Given a list of words, What would the best and easy to implement algorithm to score each word based on the word ...
0
votes
1answer
34 views

NER and context mapping

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and has to be executed within 1 ...
0
votes
0answers
523 views

Getting TypeError: float() argument must be a string or a number, not 'list' Error

I am trying to run some code and I am getting a weird error, here is the code: ...
0
votes
1answer
63 views

How can I find synonyms and antonyms for a word?

I found some code online where I can feed in a word, and find both synonyms and antonyms for this word. The code below does just that. ...
0
votes
1answer
48 views

Where does spacy, nltk, prodigy, sklearn fit in the AI project?

Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:
0
votes
0answers
62 views

find string similarity in python NLTK

Supoese I have two or more strings string1="Tell me the claim value"-- these word has to be find out string2=" This column tells us regarding the indoemation about claim" I have to find out the ...
0
votes
0answers
31 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
0
votes
0answers
19 views

Error while using NLTK/ How should I read paragraph using NLTK?

I am a total newbie. I have a custom text document which is a paragraph. I am trying to read it and display using NLTK. following is my code; ...
0
votes
0answers
19 views

KDTree classified as AI /ML?

We are currently using a KDTree to find similar documents based on keywords identified via TF-IDF. After the tree is built, we are querying the tree to find nearest points. Would this fall in the ...
0
votes
0answers
141 views

shallow parsing, dependency parsing, deep parsing

What is the difference between shallow parsing, dependency parsing, deep parsing? I read it on google search that dependency parsing is the bridge between sahallow and deep parsing. Also, the chunking ...
0
votes
2answers
245 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
0
votes
2answers
390 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
-1
votes
1answer
411 views

What is “Interpolated Absolute Discounting” smoothing method

I'm asked to implement "Interpolated Absolute Discounting" for a bigram language model for a text. First, I don't know what is it exactly. I guess it is an interpolation between different ngrams (...
-1
votes
1answer
462 views

I want to make a text extractor for extracting particular type of text from unstructured text data.

I have a lot of data and manually extracted annotations for the text. I was looking for any advice to automate the annotation extraction up to a good level of accuracy. Any advice is welcome.