Questions tagged [nltk]

NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Filter by
Sorted by
Tagged with
1
vote
2answers
430 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
3
votes
6answers
2k views

Sentiment Analysis of Movie Reviews using Python

I am currently doing sentiment analysis using Python. Here I am taking all the reviews from movie dataset and using Naive Bayes algorithm to predict whether the review is positive or negative. From ...
0
votes
1answer
66 views

Trying to compress text with NLP

For a university project, I need to send text in Spanish via SMS. As these have a cost, I am trying to compress this text in an inefficient way. This consists of first generating a permutation of ...
2
votes
1answer
468 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
0
votes
0answers
7 views

chat bot nfL6.json training data

I am experimenting with chatterbot and the nfL6.json training data set. The data looks like this below, should I be using an nbestanswers or ...
1
vote
1answer
16 views

Converting to lowercase while creating dataset for NER using spacy

I am trying to make a custom entity model for an NER application using spacy. In several NLP projects, I have converted all the data to lowercase and applied several ML techniques. For NER also should ...
0
votes
0answers
18 views

Keyword Imputation Using Machine Learning and NLP

I am fairly new to machine learning and data imputation so forgive me if I am using wrong words or not feasible ideas. I have a table as follows, with text in the Headline column and a list of ...
0
votes
1answer
233 views

How can I find synonyms and antonyms for a word?

I found some code online where I can feed in a word, and find both synonyms and antonyms for this word. The code below does just that. ...
1
vote
1answer
563 views

Elbow method for cosine distance

I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
1
vote
2answers
54 views

Need help to increase classification accuracy for classified ads posting

I have to predict the category under which ad was posted using the provided data; I cannot gain accuracy more than 74% for my model. I am not sure what I am missing. What I have done so far: Cleaned ...
0
votes
2answers
266 views

nltk.corpus for data science related words?

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found: ...
3
votes
1answer
810 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
1
vote
1answer
36 views

How do I split contents in a text that would include two or more different themes (context) in NLP?

For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning" the output should be: The airline companies have affected ...
1
vote
1answer
159 views

train NER using NLTK with custom corpora (non-english) must use StanfordNER?

I have searched about customization NER corpora for trainig the model using NLTK library from python, but all of the answer direct to nltk book chapter 7 and honestly makes me confuse how to train the ...
2
votes
1answer
173 views

Weighting of words in lexicon based sentiment analysis

I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following: $$ Score = \frac{\sum_{i}{word_i}...
1
vote
2answers
692 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
0
votes
0answers
14 views

Adding additional classes in stanford NLP NER or Spacy

For stanford NER 3 class model, Location, Person, Organization recognizers are available. Is it possible to add additional classes to this model. For example : Sports as one class to tag sports names. ...
0
votes
0answers
24 views

Stemming/lemmatization for German words

I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
0
votes
0answers
21 views

Replacing in german text "Sie" with "Du" and changing also the sentence

I have a lot of german text like Examples "Stelle Sie die Pflanze an einen sonnigen Platz." "Achten Sie bei der Umstellung darauf, das bisherige Präparat allmählich durch NEWPRODUCT zu ...
-1
votes
1answer
501 views

Building a text extractor to extract particular type of text from unstructured text data

I have a lot of data and manually extracted annotations for the text. I was looking for any advice to automate the annotation extraction up to a good level of accuracy. Any advice is welcome.
1
vote
0answers
163 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
0
votes
1answer
29 views

WordNetLemmatizer not lemmatizing the word "promotional" even with POS given

When I do wnl.lemmatize('promotional','a') or wnl.lemmatize('promotional',wordnet.ADJ), I get merely ...
0
votes
1answer
106 views

Where does spacy, nltk, prodigy, sklearn fit in the AI project?

Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:
1
vote
1answer
56 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
2
votes
0answers
129 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
2
votes
4answers
3k views

Text Classifier with multiple bag-of-words

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the ...
3
votes
1answer
55 views

NER and context mapping

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and has to be executed within 1 ...
1
vote
0answers
15 views

How to find the related columns between column?

Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike. ...
4
votes
3answers
4k views
2
votes
2answers
190 views

What can be done so that 'teacher' and 'teaches' are treated similar?

Who teaches English? Now, after tokenizing, stemming.. it gives me Who, teach, English In my list of word, I have a ...
17
votes
6answers
68k views

Similarity between two words

I'm looking for a Python library that helps me identify the similarity between two words or sentences. I will be doing Audio to Text conversion which will result in an English dictionary or non ...
4
votes
1answer
832 views

Is there an NLP corpus that contains common medical terms?

I am trying to use the NLTK library to extract keywords denoting medical symptoms from medical reports of patients. For example, I have a medical report as follows: s:a 33 year old female ...
0
votes
5answers
26k views

remove special character in a List or String

Input_String is Text_Corpus of Jane Austen Book output Should be : ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question'] But getting this Output : ['to', 'be,', 'or', 'not', 'to', '...
3
votes
2answers
190 views

Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?

For example: for a parameter like input voltage, Alias names : V_INPUT, VIN etc. Now, I want the software to recognize each of the alias names as same. Is ...
2
votes
0answers
419 views

improve NER model accuracy with spaCy dependency tree

I have search at lot, was not able to find a solution for my problem... I am training a NER model, that should detect two types of words: Instructions and Conditions. This is not the standard use-case ...
5
votes
3answers
140 views

Chunking Sentences with Spacy

I have a lot of sentences (500k) which looks like this: ...
2
votes
1answer
43 views

What is the more natural parsing, the one that leads to the preferred reading of the sentence

I have those rules: and those two possible parse trees: I am asked for the next question: What is the more natural parsing, the one that leads to the preferred reading of the sentence? Can anyone ...
1
vote
0answers
50 views

For short sentences(max length 10 ), which Name entity recognition algorithm is good?

My Training data look like this . I have to recognize 4 class for each sentence. Any algorithm , which have some learning parameters Means not rule based approach . So which method is good for my ...
2
votes
2answers
50 views

Summing three lexicon based approach methods for sentiment analysis?

I'm doing sentiment analysis using a lexicon based approach and I have a bunch of news headlines that needs to be categorized as negative, positive and neutral or within a scale ranging from -1 (very ...
3
votes
1answer
147 views

Smart sentence segmentation not splitting on abbreviations

Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the ...
0
votes
3answers
2k views

Python Code to find the number of hapax legomena in a Text or Words_List

In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, either in the written record of an entire language, in the works of an author, or in a single text. The term ...
6
votes
3answers
3k views

Training NLP with multiple text input features

Question: How can I train a NLP model with discrete labels that is based on multiple text input features? Background: I'm trying to predict the difficulty of a 4-option multiple choice exam ...
1
vote
1answer
159 views

How to properly compare these two confusion matrix?

I have used Vader, a sentiment analysis tool for social media, on a database of movie reviews. These two confusion matrices differ in the vader.py algorithm, as the first one is from nltk: The second ...
1
vote
1answer
2k views

Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a ...
0
votes
1answer
624 views

Is there a synset for phrasal verbs?

Is there a database of phrasal verbs of similar semantics? Eg. one where querying ‘get in touch’ would return ‘get in contact with’, among other phrasal verbs of similar meanings? If not, given a ...
0
votes
1answer
64 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
0
votes
1answer
115 views

BLEU_SCORE gives bad scores - what am I doing wrong?

I want to calculate the BLEU_SCORE but it gives me bad results I don't know why? For example, this is my reference and predicted sentences: ...
5
votes
3answers
1k views

Machine learning or NLP approach to convert string about month ,year into dates

I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates. Example : last year last month into December 2018 string may ...
28
votes
5answers
35k views

How can I get a measure of the semantic similarity of words?

What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal: ...
3
votes
2answers
91 views

How to automatically find the sentiment?

I have written a program that scrapes data from the web and I have in possession about 5k sentences which I want to analyze. Part 1: I am just starting out in data science and wanted to know if ...