Questions tagged [nltk]

NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

40 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
1answer
751 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
3
votes
1answer
454 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
3
votes
0answers
106 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
2
votes
1answer
35 views

How do I split contents in a text that would include two or more different themes (context) in NLP?

For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning" the output should be: The airline companies have affected ...
2
votes
0answers
106 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
2
votes
0answers
349 views

improve NER model accuracy with spaCy dependency tree

I have search at lot, was not able to find a solution for my problem... I am training a NER model, that should detect two types of words: Instructions and Conditions. This is not the standard use-case ...
2
votes
1answer
139 views

Weighting of words in lexicon based sentiment analysis

I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following: $$ Score = \frac{\sum_{i}{word_i}...
2
votes
0answers
47 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
2
votes
0answers
210 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
2
votes
0answers
174 views

Data categorization

I'm using Google API to categorize and predict problems with laptops (Hardware, Software, Network, Customer satisfaction, No reason). I inserted my training data and categories are very well evaluated ...
1
vote
2answers
52 views

Need help to increase classification accuracy for classified ads posting

I have to predict the category under which ad was posted using the provided data; I cannot gain accuracy more than 74% for my model. I am not sure what I am missing. What I have done so far: Cleaned ...
1
vote
0answers
14 views

Adding additional classes in stanford NLP NER or Spacy

For stanford NER 3 class model, Location, Person, Organization recognizers are available. Is it possible to add additional classes to this model. For example : Sports as one class to tag sports names. ...
1
vote
0answers
15 views

How to find the related columns between column?

Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike. ...
1
vote
1answer
123 views

train NER using NLTK with custom corpora (non-english) must use StanfordNER?

I have searched about customization NER corpora for trainig the model using NLTK library from python, but all of the answer direct to nltk book chapter 7 and honestly makes me confuse how to train the ...
1
vote
0answers
39 views

For short sentences(max length 10 ), which Name entity recognition algorithm is good?

My Training data look like this . I have to recognize 4 class for each sentence. Any algorithm , which have some learning parameters Means not rule based approach . So which method is good for my ...
1
vote
0answers
54 views

Semi-Supervised Learning using NLP

I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also ...
1
vote
0answers
491 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
1
vote
0answers
19 views

Determining topic of text

I was wondering what I should be looking into if I want to measure the similarity between a paragraph and a corpus of text. For example, given a paragraph of text and the entire corpus of Data ...
1
vote
1answer
516 views

Elbow method for cosine distance

I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
1
vote
2answers
240 views

nltk.corpus for data science related words?

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found: ...
1
vote
0answers
75 views

How can I find colors in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
1
vote
2answers
413 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
0answers
157 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
1
vote
0answers
841 views

How can I calculate perplexity for a bigram model?

I didn't find any function in nltk to calculate the perplexity. There are some codes I found: ...
1
vote
0answers
103 views

Feature matrix for email classification:

This is in continuation of my earlier post. In my previous model, I used just two features which couldn't fare well and gave 71% prediction (accuracy) score. Now, I'm trying to consider another ...
1
vote
0answers
833 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
25 views

Automating scoring of answers for a given question

There is a topic 'X' on which students are asked to write an English passage. Given a question, I have multiple solutions with their scorings out of 10. For a given answer by a new candidate, one ...
1
vote
2answers
656 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
1
vote
0answers
39 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
1
vote
0answers
182 views

How to dowload Wikileaks Cable Leaks documents as text corpus?

I'd like to perform NLP analysis on Wikileaks US Diplomatic Cable Leaks documents (https://wikileaks.org/plusd/), preferably as Python's NLTK3 corpus od Mongo DB documents. I couldn't find any option ...
1
vote
0answers
256 views

NLP : What are some common verbs surrounding organization names in text

I am trying to come up with some rules to detect named entities, specifically company or organization names in text. I think it makes sense to focus on verbs. There are a lot of ...
0
votes
0answers
8 views

Converting to lowercase while creating dataset for NER using spacy

I am trying to make a custom entity model for an NER application using spacy. In several NLP projects, I have converted all the data to lowercase and applied several ML techniques. For NER also should ...
0
votes
0answers
11 views

Stemming/lemmatization for German words

I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
0
votes
0answers
21 views

Replacing in german text "Sie" with "Du" and changing also the sentence

I have a lot of german text like Examples "Stelle Sie die Pflanze an einen sonnigen Platz." "Achten Sie bei der Umstellung darauf, das bisherige Präparat allmählich durch NEWPRODUCT zu ...
0
votes
1answer
50 views

Trying to compress text with NLP

For a university project, I need to send text in Spanish via SMS. As these have a cost, I am trying to compress this text in an inefficient way. This consists of first generating a permutation of ...
0
votes
1answer
201 views

How can I find synonyms and antonyms for a word?

I found some code online where I can feed in a word, and find both synonyms and antonyms for this word. The code below does just that. ...
0
votes
0answers
33 views

Error while using NLTK/ How should I read paragraph using NLTK?

I am a total newbie. I have a custom text document which is a paragraph. I am trying to read it and display using NLTK. following is my code; ...
0
votes
0answers
264 views

NLP how to go beyond simple intent finding--using context and targeting objects

I'm manually applying NLP rules to a chatbot. Currently, I've a simple set of rules--actions that follow certain trigger words. Ex: "Create the match on saturday." This has been working for ...
-1
votes
1answer
564 views

What is "Interpolated Absolute Discounting" smoothing method

I'm asked to implement "Interpolated Absolute Discounting" for a bigram language model for a text. First, I don't know what is it exactly. I guess it is an interpolation between different ngrams (...
-1
votes
1answer
499 views

Building a text extractor to extract particular type of text from unstructured text data

I have a lot of data and manually extracted annotations for the text. I was looking for any advice to automate the annotation extraction up to a good level of accuracy. Any advice is welcome.