Questions tagged [nltk]

The Natural Language Toolkit is a Python library for computational linguistics.

Filter by
Sorted by
Tagged with
1
vote
1answer
33 views

How to create a table to display relative frequencies of selected words (eg. with, can, will) from any text corpus in nltk package in python [closed]

Interested words are ['with', 'can', 'will'] gutenberg corpus is the corpus we would like to search on. Expected output is ...
0
votes
0answers
15 views

Is there a way to find difference between topics in two languages using nlp

I want to analyse queries and their differences between two different languages English and Spanish in this case. I'm aware about topic modelling. I'm in search of any corpus available or any ...
1
vote
0answers
21 views

How to group news by topics when they do not have labels

I have a dataset that includes 20000 news titles and 9 columns that are not relevant (boolean values) for labelling them. I have tried to extract top words using tokenizetion and tf-idf for n-grams. ...
1
vote
1answer
23 views

How to add a new column with labels in a dataframe?

I have thousands of sentences that I would need to label based on their sentiment. An example is ...
1
vote
1answer
27 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
1
vote
1answer
32 views

Which insights a data scientist could derive from text-analysis? [closed]

I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging. Since it is my first ...
0
votes
1answer
21 views

Text analysis: structure and sentiment

I would need to analyse the structure of texts like this: ...
0
votes
1answer
36 views

BLEU_SCORE gives bad scores - what am I doing wrong?

I want to calculate the BLEU_SCORE but it gives me bad results I don't know why? For example, this is my reference and predicted sentences: ...
2
votes
1answer
20 views

Weighting of words in lexicon based sentiment analysis

I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following: $$ Score = \frac{\sum_{i}{word_i}...
0
votes
0answers
20 views

English Word popularity scoring

I'm new to data science and looking for the word popularity ranking algorithm. Given a list of words, What would the best and easy to implement algorithm to score each word based on the word ...
0
votes
1answer
34 views

NER and context mapping

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and has to be executed within 1 ...
1
vote
1answer
34 views

How can I tokenize a text file with BERT or something similar?

I want to use the twitter datasets in a project and the tweet contents look something like this: ...
0
votes
0answers
529 views

Getting TypeError: float() argument must be a string or a number, not 'list' Error

I am trying to run some code and I am getting a weird error, here is the code: ...
1
vote
0answers
22 views

Semi-Supervised Learning using NLP

I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also ...
0
votes
1answer
63 views

How can I find synonyms and antonyms for a word?

I found some code online where I can feed in a word, and find both synonyms and antonyms for this word. The code below does just that. ...
0
votes
1answer
48 views

Where does spacy, nltk, prodigy, sklearn fit in the AI project?

Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:
0
votes
0answers
62 views

find string similarity in python NLTK

Supoese I have two or more strings string1="Tell me the claim value"-- these word has to be find out string2=" This column tells us regarding the indoemation about claim" I have to find out the ...
2
votes
1answer
286 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
1
vote
2answers
224 views

Difference between packaged sentiment analysis tools (TextBlob/NLTK) and training your own classifier?

I'm new to ML and training classifiers in practice, so I was just wondering what the difference was between the built-in sentiment tools of packages such as NLTK and TextBlob as compared to manually ...
1
vote
3answers
200 views

Detect if word is «common English» word or slang word

I have a huge list of short phrases, for example: ...
5
votes
2answers
46 views

Chunking Sentences with Spacy

I have a lot of sentences (500k) which looks like this: ...
0
votes
1answer
107 views

NLP techniques to label unlabeled data in a dataset [closed]

So I have a .xls file with negative and neutral reviews of a medicine. However, this dataset does not have labels. I converted this .xls into a dataframe and I am using the Spacy Lib. I have to use ...
0
votes
0answers
31 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
4
votes
3answers
352 views

TFIDF for very short sentences

I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's ...
1
vote
1answer
15 views

nltk measure the accuracy of the new features

I have been playing around the NLTK algorithm for some data prediction. Starting from this gib, I started my understanding process. However, there are some bits that don't make sense. If I have a ...
1
vote
0answers
156 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
1
vote
1answer
34 views

How do I use NLTK Text object with Re library?

I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error; ...
0
votes
0answers
19 views

Error while using NLTK/ How should I read paragraph using NLTK?

I am a total newbie. I have a custom text document which is a paragraph. I am trying to read it and display using NLTK. following is my code; ...
0
votes
0answers
19 views

KDTree classified as AI /ML?

We are currently using a KDTree to find similar documents based on keywords identified via TF-IDF. After the tree is built, we are querying the tree to find nearest points. Would this fall in the ...
1
vote
1answer
93 views
2
votes
0answers
40 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
2
votes
1answer
738 views

Is there a good German Stemmer?

What I tried: ...
0
votes
1answer
44 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
1
vote
0answers
19 views

Determining topic of text

I was wondering what I should be looking into if I want to measure the similarity between a paragraph and a corpus of text. For example, given a paragraph of text and the entire corpus of Data ...
0
votes
1answer
128 views

Text PreProcessing not improving accuracy

I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs ...
0
votes
1answer
96 views

NLTK pos-tag fails to recognize names

I'm trying to identify all the names in a novel (fed as a text file) using NLTK. On a smaller scale, the POS tagging works perfectly. However, when I feed a large body of text, by which I mean three ...
1
vote
1answer
275 views

Elbow method for cosine distance

I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
1
vote
2answers
838 views

Which libraries in Python are there in NLP to tokenize the Hindi sentence?

For English language there are libraries like NLTK, CoreNLP which are used for Text Normalization, Word Tokenization and Detokenization, Sentence Splitting etc. Like English, is there any library to ...
0
votes
1answer
2k views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
0
votes
1answer
181 views

Extracting email id gives error

I am extracting email ids and storing them into a new column variable, but I am getting the issue: ...
1
vote
2answers
126 views

nltk.corpus for data science related words?

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found: ...
1
vote
1answer
387 views

From most frequent words how to extract technical skill words

I've scrape 30 job description web and stored them into a list called job_desc where each item is a job description. ...
1
vote
0answers
42 views

How can I find colors in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
2
votes
1answer
246 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
0
votes
2answers
245 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
0answers
140 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
1
vote
1answer
510 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
1
vote
2answers
55 views

Doc2vec '-' symbol occurrence

Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'. Vector representations of these words are incorrectly classified by doc2vec. ['...
-1
votes
1answer
411 views

What is “Interpolated Absolute Discounting” smoothing method

I'm asked to implement "Interpolated Absolute Discounting" for a bigram language model for a text. First, I don't know what is it exactly. I guess it is an interpolation between different ngrams (...
1
vote
0answers
417 views

How can I calculate perplexity for a bigram model?

I didn't find any function in nltk to calculate the perplexity. There are some codes I found: ...