Questions tagged [nltk]

NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Keyword Imputation Using Machine Learning and NLP

I am fairly new to machine learning and data imputation so forgive me if I am using wrong words or not feasible ideas. I have a table as follows, with text in the Headline column and a list of ...
0
votes
0answers
21 views

Question generation from a wordcloud using nlp [closed]

Need help in Question generation from Clickable Image, Wordcloud,Upvote - Downvote,Read and Response Type using NLP
0
votes
0answers
11 views

Converting to lowercase while creating dataset for NER using spacy

I am trying to make a custom entity model for an NER application using spacy. In several NLP projects, I have converted all the data to lowercase and applied several ML techniques. For NER also should ...
1
vote
2answers
54 views

Need help to increase classification accuracy for classified ads posting

I have to predict the category under which ad was posted using the provided data; I cannot gain accuracy more than 74% for my model. I am not sure what I am missing. What I have done so far: Cleaned ...
1
vote
0answers
14 views

Adding additional classes in stanford NLP NER or Spacy

For stanford NER 3 class model, Location, Person, Organization recognizers are available. Is it possible to add additional classes to this model. For example : Sports as one class to tag sports names. ...
0
votes
0answers
17 views

Stemming/lemmatization for German words

I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
0
votes
0answers
21 views

Replacing in german text "Sie" with "Du" and changing also the sentence

I have a lot of german text like Examples "Stelle Sie die Pflanze an einen sonnigen Platz." "Achten Sie bei der Umstellung darauf, das bisherige Präparat allmählich durch NEWPRODUCT zu ...
0
votes
1answer
56 views

Trying to compress text with NLP

For a university project, I need to send text in Spanish via SMS. As these have a cost, I am trying to compress this text in an inefficient way. This consists of first generating a permutation of ...
0
votes
1answer
28 views

WordNetLemmatizer not lemmatizing the word "promotional" even with POS given

When I do wnl.lemmatize('promotional','a') or wnl.lemmatize('promotional',wordnet.ADJ), I get merely ...
1
vote
0answers
15 views

How to find the related columns between column?

Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike. ...
2
votes
1answer
35 views

How do I split contents in a text that would include two or more different themes (context) in NLP?

For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning" the output should be: The airline companies have affected ...
2
votes
0answers
123 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
4
votes
1answer
683 views

Is there an NLP corpus that contains common medical terms?

I am trying to use the NLTK library to extract keywords denoting medical symptoms from medical reports of patients. For example, I have a medical report as follows: s:a 33 year old female ...
3
votes
2answers
189 views

Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?

For example: for a parameter like input voltage, Alias names : V_INPUT, VIN etc. Now, I want the software to recognize each of the alias names as same. Is ...
1
vote
1answer
139 views

train NER using NLTK with custom corpora (non-english) must use StanfordNER?

I have searched about customization NER corpora for trainig the model using NLTK library from python, but all of the answer direct to nltk book chapter 7 and honestly makes me confuse how to train the ...
2
votes
1answer
43 views

What is the more natural parsing, the one that leads to the preferred reading of the sentence

I have those rules: and those two possible parse trees: I am asked for the next question: What is the more natural parsing, the one that leads to the preferred reading of the sentence? Can anyone ...
2
votes
0answers
389 views

improve NER model accuracy with spaCy dependency tree

I have search at lot, was not able to find a solution for my problem... I am training a NER model, that should detect two types of words: Instructions and Conditions. This is not the standard use-case ...
1
vote
0answers
45 views

For short sentences(max length 10 ), which Name entity recognition algorithm is good?

My Training data look like this . I have to recognize 4 class for each sentence. Any algorithm , which have some learning parameters Means not rule based approach . So which method is good for my ...
2
votes
2answers
50 views

Summing three lexicon based approach methods for sentiment analysis?

I'm doing sentiment analysis using a lexicon based approach and I have a bunch of news headlines that needs to be categorized as negative, positive and neutral or within a scale ranging from -1 (very ...
3
votes
1answer
132 views

Smart sentence segmentation not splitting on abbreviations

Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the ...
1
vote
1answer
147 views

How to properly compare these two confusion matrix?

I have used Vader, a sentiment analysis tool for social media, on a database of movie reviews. These two confusion matrices differ in the vader.py algorithm, as the first one is from nltk: The second ...
1
vote
1answer
128 views

How to create a table to display relative frequencies of selected words (eg. with, can, will) from any text corpus in nltk package in python [closed]

Interested words are ['with', 'can', 'will'] gutenberg corpus is the corpus we would like to search on. Expected output is ...
1
vote
1answer
135 views

How to add a new column with labels in a dataframe?

I have thousands of sentences that I would need to label based on their sentiment. An example is ...
1
vote
1answer
54 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
1
vote
1answer
34 views

Which insights a data scientist could derive from text-analysis? [closed]

I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging. Since it is my first ...
0
votes
1answer
23 views

Text analysis: structure and sentiment

I would need to analyse the structure of texts like this: ...
0
votes
1answer
104 views

BLEU_SCORE gives bad scores - what am I doing wrong?

I want to calculate the BLEU_SCORE but it gives me bad results I don't know why? For example, this is my reference and predicted sentences: ...
2
votes
1answer
167 views

Weighting of words in lexicon based sentiment analysis

I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following: $$ Score = \frac{\sum_{i}{word_i}...
3
votes
1answer
55 views

NER and context mapping

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and has to be executed within 1 ...
1
vote
1answer
80 views

How can I tokenize a text file with BERT or something similar?

I want to use the twitter datasets in a project and the tweet contents look something like this: ...
1
vote
0answers
56 views

Semi-Supervised Learning using NLP

I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also ...
0
votes
1answer
223 views

How can I find synonyms and antonyms for a word?

I found some code online where I can feed in a word, and find both synonyms and antonyms for this word. The code below does just that. ...
0
votes
1answer
99 views

Where does spacy, nltk, prodigy, sklearn fit in the AI project?

Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:
3
votes
1answer
796 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
1
vote
2answers
565 views

Difference between packaged sentiment analysis tools (TextBlob/NLTK) and training your own classifier?

I'm new to ML and training classifiers in practice, so I was just wondering what the difference was between the built-in sentiment tools of packages such as NLTK and TextBlob as compared to manually ...
1
vote
3answers
802 views

Detect if word is «common English» word or slang word

I have a huge list of short phrases, for example: ...
5
votes
3answers
123 views

Chunking Sentences with Spacy

I have a lot of sentences (500k) which looks like this: ...
0
votes
1answer
256 views

NLP techniques to label unlabeled data in a dataset [closed]

So I have a .xls file with negative and neutral reviews of a medicine. However, this dataset does not have labels. I converted this .xls into a dataframe and I am using the Spacy Lib. I have to use ...
4
votes
3answers
1k views

TFIDF for very short sentences

I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's ...
1
vote
1answer
19 views

nltk measure the accuracy of the new features

I have been playing around the NLTK algorithm for some data prediction. Starting from this gib, I started my understanding process. However, there are some bits that don't make sense. If I have a ...
1
vote
0answers
497 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
1
vote
1answer
136 views

How do I use NLTK Text object with Re library?

I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error; ...
0
votes
0answers
35 views

Error while using NLTK/ How should I read paragraph using NLTK?

I am a total newbie. I have a custom text document which is a paragraph. I am trying to read it and display using NLTK. following is my code; ...
1
vote
1answer
197 views
2
votes
0answers
50 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
4
votes
3answers
4k views

Is there a good German Stemmer?

What I tried: ...
0
votes
1answer
62 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
1
vote
0answers
19 views

Determining topic of text

I was wondering what I should be looking into if I want to measure the similarity between a paragraph and a corpus of text. For example, given a paragraph of text and the entire corpus of Data ...
0
votes
1answer
158 views

Text PreProcessing not improving accuracy

I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs ...
0
votes
1answer
221 views

NLTK pos-tag fails to recognize names

I'm trying to identify all the names in a novel (fed as a text file) using NLTK. On a smaller scale, the POS tagging works perfectly. However, when I feed a large body of text, by which I mean three ...