Questions tagged [nltk]

NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Filter by
Sorted by
Tagged with
1
vote
2answers
406 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
0
votes
1answer
42 views

Trying to compress text with NLP

For a university project, I need to send text in Spanish via SMS. As these have a cost, I am trying to compress this text in an inefficient way. This consists of first generating a permutation of ...
1
vote
0answers
12 views

Adding additional classes in stanford NLP NER or Spacy

For stanford NER 3 class model, Location, Person, Organization recognizers are available. Is it possible to add additional classes to this model. For example : Sports as one class to tag sports names. ...
0
votes
0answers
9 views

Stemming/lemmatization for German words

I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
3
votes
1answer
442 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
0
votes
0answers
19 views

Replacing in german text "Sie" with "Du" and changing also the sentence

I have a lot of german text like Examples "Stelle Sie die Pflanze an einen sonnigen Platz." "Achten Sie bei der Umstellung darauf, das bisherige Präparat allmählich durch NEWPRODUCT zu ...
-1
votes
1answer
499 views

Building a text extractor to extract particular type of text from unstructured text data

I have a lot of data and manually extracted annotations for the text. I was looking for any advice to automate the annotation extraction up to a good level of accuracy. Any advice is welcome.
1
vote
0answers
155 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
0
votes
1answer
187 views

How can I find synonyms and antonyms for a word?

I found some code online where I can feed in a word, and find both synonyms and antonyms for this word. The code below does just that. ...
1
vote
1answer
488 views

Elbow method for cosine distance

I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
0
votes
1answer
24 views

WordNetLemmatizer not lemmatizing the word "promotional" even with POS given

When I do wnl.lemmatize('promotional','a') or wnl.lemmatize('promotional',wordnet.ADJ), I get merely ...
0
votes
1answer
89 views

Where does spacy, nltk, prodigy, sklearn fit in the AI project?

Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:
1
vote
1answer
51 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
1
vote
2answers
234 views

nltk.corpus for data science related words?

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found: ...
3
votes
1answer
732 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
2
votes
0answers
96 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
1
vote
1answer
99 views

train NER using NLTK with custom corpora (non-english) must use StanfordNER?

I have searched about customization NER corpora for trainig the model using NLTK library from python, but all of the answer direct to nltk book chapter 7 and honestly makes me confuse how to train the ...
1
vote
1answer
33 views

How do I split contents in a text that would include two or more different themes (context) in NLP?

For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning" the output should be: The airline companies have affected ...
2
votes
4answers
3k views

Text Classifier with multiple bag-of-words

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the ...
2
votes
1answer
101 views

Weighting of words in lexicon based sentiment analysis

I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following: $$ Score = \frac{\sum_{i}{word_i}...
3
votes
1answer
53 views

NER and context mapping

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and has to be executed within 1 ...
1
vote
2answers
639 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
1
vote
0answers
15 views

How to find the related columns between column?

Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike. ...
4
votes
3answers
3k views
2
votes
2answers
187 views

What can be done so that 'teacher' and 'teaches' are treated similar?

Who teaches English? Now, after tokenizing, stemming.. it gives me Who, teach, English In my list of word, I have a ...
17
votes
6answers
65k views

Similarity between two words

I'm looking for a Python library that helps me identify the similarity between two words or sentences. I will be doing Audio to Text conversion which will result in an English dictionary or non ...
4
votes
1answer
419 views

Is there an NLP corpus that contains common medical terms?

I am trying to use the NLTK library to extract keywords denoting medical symptoms from medical reports of patients. For example, I have a medical report as follows: s:a 33 year old female ...
0
votes
5answers
26k views

remove special character in a List or String

Input_String is Text_Corpus of Jane Austen Book output Should be : ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question'] But getting this Output : ['to', 'be,', 'or', 'not', 'to', '...
3
votes
2answers
179 views

Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?

For example: for a parameter like input voltage, Alias names : V_INPUT, VIN etc. Now, I want the software to recognize each of the alias names as same. Is ...
2
votes
0answers
309 views

improve NER model accuracy with spaCy dependency tree

I have search at lot, was not able to find a solution for my problem... I am training a NER model, that should detect two types of words: Instructions and Conditions. This is not the standard use-case ...
5
votes
3answers
86 views

Chunking Sentences with Spacy

I have a lot of sentences (500k) which looks like this: ...
2
votes
1answer
43 views

What is the more natural parsing, the one that leads to the preferred reading of the sentence

I have those rules: and those two possible parse trees: I am asked for the next question: What is the more natural parsing, the one that leads to the preferred reading of the sentence? Can anyone ...
1
vote
0answers
38 views

For short sentences(max length 10 ), which Name entity recognition algorithm is good?

My Training data look like this . I have to recognize 4 class for each sentence. Any algorithm , which have some learning parameters Means not rule based approach . So which method is good for my ...
2
votes
2answers
50 views

Summing three lexicon based approach methods for sentiment analysis?

I'm doing sentiment analysis using a lexicon based approach and I have a bunch of news headlines that needs to be categorized as negative, positive and neutral or within a scale ranging from -1 (very ...
3
votes
1answer
92 views

Smart sentence segmentation not splitting on abbreviations

Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the ...
0
votes
3answers
2k views

Python Code to find the number of hapax legomena in a Text or Words_List

In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, either in the written record of an entire language, in the works of an author, or in a single text. The term ...
6
votes
3answers
3k views

Training NLP with multiple text input features

Question: How can I train a NLP model with discrete labels that is based on multiple text input features? Background: I'm trying to predict the difficulty of a 4-option multiple choice exam ...
1
vote
1answer
124 views

How to properly compare these two confusion matrix?

I have used Vader, a sentiment analysis tool for social media, on a database of movie reviews. These two confusion matrices differ in the vader.py algorithm, as the first one is from nltk: The second ...
1
vote
1answer
2k views

Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a ...
0
votes
1answer
584 views

Is there a synset for phrasal verbs?

Is there a database of phrasal verbs of similar semantics? Eg. one where querying ‘get in touch’ would return ‘get in contact with’, among other phrasal verbs of similar meanings? If not, given a ...
0
votes
1answer
59 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
0
votes
1answer
96 views

BLEU_SCORE gives bad scores - what am I doing wrong?

I want to calculate the BLEU_SCORE but it gives me bad results I don't know why? For example, this is my reference and predicted sentences: ...
5
votes
3answers
1k views

Machine learning or NLP approach to convert string about month ,year into dates

I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates. Example : last year last month into December 2018 string may ...
27
votes
5answers
33k views

How can I get a measure of the semantic similarity of words?

What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal: ...
3
votes
2answers
90 views

How to automatically find the sentiment?

I have written a program that scrapes data from the web and I have in possession about 5k sentences which I want to analyze. Part 1: I am just starting out in data science and wanted to know if ...
3
votes
2answers
1k views

Word analysis in Python

I have a list of documents which look like this: ...
1
vote
1answer
462 views

sentiment analysis nltk python

I'd like to perform sentiment analysis on stock comment using scikit and nltk. I already have about 100 comments on different stocks like "this stock will rock" which I marked as positive (1) or "this ...
1
vote
1answer
112 views

How to create a table to display relative frequencies of selected words (eg. with, can, will) from any text corpus in nltk package in python [closed]

Interested words are ['with', 'can', 'will'] gutenberg corpus is the corpus we would like to search on. Expected output is ...
7
votes
2answers
2k views

Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis

As a part of my university project, I am researching/developing a sentiment analysis model wherein I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning ...
0
votes
1answer
23 views

Text analysis: structure and sentiment

I would need to analyse the structure of texts like this: ...