Questions tagged [nltk]
NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.
138 questions
1
vote
1
answer
107
views
Number of stop words variation in libaries sklearn and nltk
Is there a reason why there is a big variation in the number of stop words? I assumed that there would be a general agreement from English experts how many stop words there could be. And even with ...
- 135
3
votes
1
answer
1k
views
Sentence tokenization for sentence without punctuation
I wish to perform sentence tokenization for sentence without punctuation, below is the code:
...
- 775
1
vote
1
answer
122
views
0
votes
1
answer
161
views
Why custom training a Spacy model runs only the Initializing pipeline but the Training pipeline is not running?
I am training a custom NER model with Spacy version 3.5.0 using some dummy data. My entire code and dummy data is given below. This is exact same code give in the 2nd half of this link. The code is ...
- 185
1
vote
1
answer
565
views
NLP Question: Where can I find a list of all open compound words (with spaces in them), like "peanut butter" or "high school"? [close
I already have a list of "1-gram" words, which include closed compound words like "skyscraper" or "weatherman."
However, I'm also interested in compiling a list of "...
- 13
3
votes
1
answer
535
views
2
votes
0
answers
38
views
Looking for a generalized (extended) lemmatizer
Whenever I lemmatize a compound word in English or German, I obtain a result that ignores the compound structure, e.g. for 'sidekicks' the NLTK WordNet lemmatizer returns 'sidekick', for '...
1
vote
0
answers
21
views
Which Pointers from WordNet are Used for Synset in NLTK
I'm trying to create a custom parser for wordnet and hit a roadblock. I see that there are tons of different pointer_symbols and lots of them seem almost like synonyms but not exactly synonyms. I'm ...
- 117
1
vote
1
answer
46
views
Natural language processing [closed]
I am new to NLP. I converted my JSON file to CSV with the Jupyter notebook. I am unsure how to proceed in pre-processing my data using techniques such as tokenization and lemmatization etc. I ...
- 11
3
votes
1
answer
277
views
Compare Books using book categories list NLP
I have a database of books. Each book has a list of categories that describe the genre/topics of the book (I use Python models).
The categories in the list most of the time are composed of 1 to 3 ...
- 131
2
votes
1
answer
871
views
Converting to lowercase while creating dataset for NER using spacy
I am trying to make a custom entity model for an NER application using spacy. In several NLP projects, I have converted all the data to lowercase and applied several ML techniques. For NER also should ...
- 193
1
vote
2
answers
114
views
Need help to increase classification accuracy for classified ads posting
I have to predict the category under which ad was posted using the provided data; I cannot gain accuracy more than 74% for my model. I am not sure what I am missing.
What I have done so far:
Cleaned ...
- 21
2
votes
1
answer
1k
views
Stemming/lemmatization for German words
I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
- 141
1
vote
1
answer
463
views
Trying to compress text with NLP
For a university project, I need to send text in Spanish via SMS. As these have a cost, I am trying to compress this text in an inefficient way. This consists of first generating a permutation of ...
- 11
0
votes
1
answer
185
views
WordNetLemmatizer not lemmatizing the word "promotional" even with POS given
When I do wnl.lemmatize('promotional','a') or wnl.lemmatize('promotional',wordnet.ADJ), I get merely ...
- 133
1
vote
0
answers
41
views
How to find the related columns between column?
Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike.
...
1
vote
1
answer
701
views
How do I split contents in a text that would include two or more different themes (context) in NLP?
For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning"
the output should be:
The airline companies have affected ...
2
votes
0
answers
298
views
Text preprocessing on corpus in pipeline before Gensim word2vec training
I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this:
...
7
votes
1
answer
5k
views
Is there an NLP corpus that contains common medical terms?
I am trying to use the NLTK library to extract keywords denoting medical symptoms from medical reports of patients. For example, I have a medical report as follows:
s:a 33 year old female ...
user112647
3
votes
2
answers
403
views
Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?
For example: for a parameter like input voltage,
Alias names : V_INPUT, VIN etc.
Now, I want the software to recognize each of the alias names as same. Is ...
- 71
1
vote
1
answer
525
views
train NER using NLTK with custom corpora (non-english) must use StanfordNER?
I have searched about customization NER corpora for trainig the model using NLTK library from python, but all of the answer direct to nltk book chapter 7 and honestly makes me confuse how to train the ...
- 41
2
votes
1
answer
54
views
What is the more natural parsing, the one that leads to the preferred reading of the sentence
I have those rules:
and those two possible parse trees:
I am asked for the next question:
What is the more natural parsing, the one that leads to the preferred reading of the sentence?
Can anyone ...
- 157
2
votes
0
answers
882
views
improve NER model accuracy with spaCy dependency tree
I have search at lot, was not able to find a solution for my problem...
I am training a NER model, that should detect two types of words: Instructions and Conditions. This is not the standard use-case ...
- 21
1
vote
0
answers
171
views
For short sentences(max length 10 ), which Name entity recognition algorithm is good?
My Training data look like this .
I have to recognize 4 class for each sentence.
Any algorithm , which have some learning parameters Means not rule based approach . So which method is good for my ...
- 11
2
votes
2
answers
119
views
Summing three lexicon based approach methods for sentiment analysis?
I'm doing sentiment analysis using a lexicon based approach and I have a bunch of news headlines that needs to be categorized as negative, positive and neutral or within a scale ranging from -1 (very ...
- 65
3
votes
1
answer
563
views
Smart sentence segmentation not splitting on abbreviations
Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the ...
- 209
1
vote
1
answer
1k
views
How to properly compare these two confusion matrix?
I have used Vader, a sentiment analysis tool for social media, on a database of movie reviews. These two confusion matrices differ in the vader.py algorithm, as the first one is from nltk:
The second ...
- 91
1
vote
1
answer
327
views
How to create a table to display relative frequencies of selected words (eg. with, can, will) from any text corpus in nltk package in python [closed]
Interested words are ['with', 'can', 'will']
gutenberg corpus is the corpus we would like to search on.
Expected output is
...
- 11
1
vote
1
answer
498
views
How to add a new column with labels in a dataframe?
I have thousands of sentences that I would need to label based on their sentiment.
An example is
...
- 165
1
vote
1
answer
97
views
Text comparison: spot the differences
I would like to know what would be the best approach to compare two texts and see the differences between them.
For example:
...
- 161
1
vote
1
answer
42
views
Which insights a data scientist could derive from text-analysis? [closed]
I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging.
Since it is my first ...
- 165
0
votes
1
answer
38
views
Text analysis: structure and sentiment
I would need to analyse the structure of texts like this:
...
- 161
0
votes
1
answer
630
views
BLEU_SCORE gives bad scores - what am I doing wrong?
I want to calculate the BLEU_SCORE but it gives me bad results I don't know why?
For example, this is my reference and predicted sentences:
...
- 103
3
votes
1
answer
478
views
Weighting of words in lexicon based sentiment analysis
I am trying to do a lexicon based sentiment analysis on my data, where I calculate the sentiment score as follows:
$$ Score = \frac{\sum_{i}{word_i}}{\mid words \mid} $$
So according to the score the ...
- 41
3
votes
1
answer
133
views
NER and context mapping
I want to extract various amounts and tenure of contracts from different contract documents that we have.
For example :
Mr xyz, this contact is valid for 3 Months and has to be executed within 1 ...
- 574
1
vote
1
answer
236
views
How can I tokenize a text file with BERT or something similar?
I want to use the twitter datasets in a project and the tweet contents look something like this:
...
- 11
1
vote
0
answers
85
views
Semi-Supervised Learning using NLP
I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also ...
- 19
0
votes
1
answer
914
views
How can I find synonyms and antonyms for a word?
I found some code online where I can feed in a word, and find both synonyms and antonyms for this word. The code below does just that.
...
- 613
0
votes
1
answer
442
views
Where does spacy, nltk, prodigy, sklearn fit in the AI project?
Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:
- 217
3
votes
1
answer
1k
views
find bigrams in pandas
I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
- 81
1
vote
2
answers
803
views
Difference between packaged sentiment analysis tools (TextBlob/NLTK) and training your own classifier?
I'm new to ML and training classifiers in practice, so I was just wondering what the difference was between the built-in sentiment tools of packages such as NLTK and TextBlob as compared to manually ...
2
votes
3
answers
2k
views
Detect if word is «common English» word or slang word
I have a huge list of short phrases, for example:
...
- 123
6
votes
3
answers
574
views
0
votes
1
answer
443
views
NLP techniques to label unlabeled data in a dataset [closed]
So I have a .xls file with negative and neutral reviews of a medicine. However, this dataset does not have labels. I converted this .xls into a dataframe and I am using the Spacy Lib.
I have to use ...
4
votes
3
answers
3k
views
TFIDF for very short sentences
I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's ...
- 303
1
vote
1
answer
87
views
nltk measure the accuracy of the new features
I have been playing around the NLTK algorithm for some data prediction.
Starting from this gib, I started my understanding process. However, there are some bits that don't make sense.
If I have a ...
- 351
1
vote
0
answers
662
views
Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe
I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words.
A dedicated function, returning a tuple, ...
- 351
1
vote
1
answer
278
views
How do I use NLTK Text object with Re library?
I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error;
...
- 169
0
votes
0
answers
52
views
Error while using NLTK/ How should I read paragraph using NLTK?
I am a total newbie. I have a custom text document which is a paragraph. I am trying to read it and display using NLTK.
following is my code;
...
- 169
1
vote
1
answer
252
views
