Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [nltk]

The Natural Language Toolkit is a Python library for computational linguistics.

Filter by
Sorted by
Tagged with
0
votes
0answers
27 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
2
votes
3answers
39 views

TFIDF for very short sentences

I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's ...
0
votes
1answer
13 views

nltk measure the accuracy of the new features

Have been playing around the NLTK algorithm for some data prediction. Starting form this gib, I started my understanding process. However, there are some bits that don't make sense. If I have a set ...
1
vote
0answers
23 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
1
vote
1answer
23 views

How do I use NLTK Text object with Re library?

I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error; ...
0
votes
0answers
14 views

Error while using NLTK/ How should I read paragraph using NLTK?

I am a total newbie. I have a custom text document which is a paragraph. I am trying to read it and display using NLTK. following is my code; ...
0
votes
0answers
17 views

KDTree classified as AI /ML?

We are currently using a KDTree to find similar documents based on keywords identified via TF-IDF. After the tree is built, we are querying the tree to find nearest points. Would this fall in the ...
0
votes
0answers
19 views

shallow parsing, dependency parsing, deep parsing

What is the difference between shallow parsing, dependency parsing, deep parsing? I read it on google search that dependency parsing is the bridge between sahallow and deep parsing. Also, the chunking ...
1
vote
1answer
27 views
2
votes
0answers
31 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
1
vote
1answer
37 views

Is there a good German Stemmer?

What I tried: ...
0
votes
1answer
28 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
1
vote
0answers
17 views

Determining topic of text

I was wondering what I should be looking into if I want to measure the similarity between a paragraph and a corpus of text. For example, given a paragraph of text and the entire corpus of Data ...
0
votes
1answer
90 views

Text PreProcessing not improving accuracy

I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs ...
0
votes
1answer
66 views

NLTK pos-tag fails to recognize names

I'm trying to identify all the names in a novel (fed as a text file) using NLTK. On a smaller scale, the POS tagging works perfectly. However, when I feed a large body of text, by which I mean three ...
1
vote
1answer
92 views

Elbow method for cosine distance

I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
0
votes
1answer
121 views

Which libraries in Python are there in NLP to tokenize the Hindi sentence?

For English language there are libraries like NLTK, CoreNLP which are used for Text Normalization, Word Tokenization and Detokenization, Sentence Splitting etc. Like English, is there any library to ...
0
votes
1answer
552 views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
0
votes
1answer
63 views

Extracting email id gives error

I am extracting email ids and storing them into a new column variable, but I am getting the issue: ...
1
vote
2answers
60 views

nltk.corpus for data science related words?

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found: ...
1
vote
1answer
136 views

From most frequent words how to extract technical skill words

I've scrape 30 job description web and stored them into a list called job_desc where each item is a job description. ...
1
vote
1answer
58 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
0
votes
2answers
68 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
0answers
116 views

Named Entity Recognition using context of the sentence

I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence. For example: If we have to extract date field which is ...
1
vote
1answer
228 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
1
vote
2answers
51 views

Doc2vec '-' symbol occurrence

Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'. Vector representations of these words are incorrectly classified by doc2vec. ['...
0
votes
0answers
32 views

How to evaluate an ngram on test data?

I created a MLE bigram language model on a text, however, I don't know how to apply it on test data: The following is my try: ...
-1
votes
1answer
216 views

What is “Interpolated Absolute Discounting” smoothing method

I'm asked to implement "Interpolated Absolute Discounting" for a bigram language model for a text. First, I don't know what is it exactly. I guess it is an interpolation between different ngrams (...
1
vote
0answers
122 views

How can I calculate perplexity for a bigram model?

I didn't find any function in nltk to calculate the perplexity. There are some codes I found: ...
0
votes
2answers
2k views

How to use bigrams for a text of sentences?

I have a text which has many sentences. How can I use nltk.ngrams to process it? This is my code: ...
2
votes
1answer
401 views

Training NLP with multiple text input features

Question: How can I train a NLP model with discrete labels that is based on multiple text input features? Background: I'm trying to predict the difficulty of a 4-option multiple choice exam ...
1
vote
0answers
98 views

Feature matrix for email classification:

This is in continuation of my earlier post. In my previous model, I used just two features which couldn't fare well and gave 71% prediction (accuracy) score. Now, I'm trying to consider another ...
3
votes
3answers
340 views

Machine learning or NLP approach to convert string about month ,year into dates

I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates. Example : last year last month into December 2018 string may ...
1
vote
1answer
78 views
1
vote
0answers
278 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
19 views

Automating scoring of answers for a given question

There is a topic 'X' on which students are asked to write an English passage. Given a question, I have multiple solutions with their scorings out of 10. For a given answer by a new candidate, one ...
1
vote
3answers
259 views

Classification: how to handle reviews/long english words in feature set with all other numerical features

I am currently working on an use case where feature set contains numeric values such as amount, as well as a review feature which contains long winded english text. the english text will very well ...
1
vote
1answer
127 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
0
votes
0answers
101 views

Sentiment analysis with nltk

I'm studying sentimental analysis with python library nltk, following this example: ...
1
vote
1answer
125 views

Sentiment analysis for multiple entry in one text

I would like to do sentiment analysis on a set of financial news from the S&P 500 for given entities (organization names). However, each news (rows in my dataset) may have more than one entity and ...
0
votes
2answers
376 views

Python Code to find the number of hapax legomena in a Text or Words_List

In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, either in the written record of an entire language, in the works of an author, or in a single text. The term ...
0
votes
5answers
5k views

remove special character in a List or String

Input_String is Text_Corpus of Jane Austen Book output Should be : ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question'] But getting this Output : ['to', 'be,', 'or', 'not', 'to', '...
0
votes
1answer
246 views

sentiment analysis nltk python

I'd like to perform sentiment analysis on stock comment using scikit and nltk. I already have about 100 comments on different stocks like "this stock will rock" which I marked as positive (1) or "this ...
2
votes
3answers
363 views

Why do we have to remove most common words for text analysis?

I am trying to do sentiment analysis the task is to classify racist tweets from other tweets. And I have read many articles and many have mentioned to remove the most common 10 words from the column ...
0
votes
1answer
640 views

Where to know the list of NLTK tagset?

I need to know what the abbreviations mean, e.g 'VBD', 'NN', etc. I am learning text processing and now I want to get verbs in the sentences, so that I need to know which abbreviations represent any ...
1
vote
0answers
195 views

NLP how to go beyond simple intent finding--using context and targeting objects

I'm manually applying NLP rules to a chatbot. Currently, I've a simple set of rules--actions that follow certain trigger words. Ex: "Create the match on saturday." This has been working for ...
0
votes
1answer
227 views

What would be the best way to map similar ngrams

I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same ...
3
votes
0answers
34 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
3
votes
0answers
86 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
1
vote
0answers
144 views

TextRank using BM25F

I am trying to incorporate BM25F in textrank, I found out a scoring module http://whoosh.readthedocs.io/en/latest/api/scoring.html, but am unable to implement it. Has anyone incorporated this in ...