Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

-2
votes
0answers
16 views

Cannot load the entire text dataset

I have downloaded the popular Enron email dataset. The dataset has two folders. When I try to import this in the Import documents widget, it skips files. Out of 5172 files it skips fifteen of them. ...
0
votes
1answer
10 views

Recurrent Neural Networks Over Multiple Documents Over Time

So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on ...
0
votes
0answers
13 views

Feeding machine learning model with different matrix

Well my question is a general question. I tried to find some relevant information before posting my question here, but no success!. I am working on ...
0
votes
1answer
14 views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...
0
votes
0answers
17 views

What is the best way to use word2vec for bilingual text similarity?

I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this: ...
1
vote
1answer
13 views

Grouping domain specific words/phrases with same meaning

I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/...
1
vote
1answer
16 views

For text classification that has innumerable features, how do I choose the number of neurons and layers for MLPClassifier?

In my use case of text classification (identify the author from a subset of 10 authors), I find that post all processing with trigrams, there are a 100 thousand and odd features with nearly 50k ...
0
votes
1answer
11 views

Composing phrases into a grammatically correct sentence?

I'm wondering if there exist any models which could take in an ordered list of phrases without punctuation and generate a grammatically correct sentence from it. For example, for the input: ["My dog"...
0
votes
0answers
9 views

TNIC scores: why would they choose only 25% of unique words

The paper described here is Gerard Hoberg and Gordon Phillips: Text-Based Network Industries and Endogenous Product Differentiation. My question is about the way they construct the bag of words for ...
1
vote
2answers
38 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
0
votes
0answers
19 views

Classification Text with fragmentation Dataset to analyze class by class

I am working with classifying texts and each class in my classification has 'similar' characteristics, that is, standards that classify it as that characteristic. I separated my Dataframe from these ...
0
votes
0answers
18 views

How to approach EDI to generate XML using machine learning?

I am a programmer and I have near zero knowledge on machine learning (except the idea of CNNs). As of this stage I was thinking whether it is possible to perform EDI mapping to XML automatically, at ...
0
votes
0answers
20 views

Evaluation of sentence clustering using pair-wise similarities

I am trying to evaluate different approaches for (text) sentence clustering. I have a dataset with (incomplete) pair-wise similarities between sentences, e.g. sentence1 | sentence2 | similarity-score ...
0
votes
0answers
21 views

How do I measure trade-off relationship of word in text mining?

Are there any method to measure trade-off relationship of word's meaning in text mining? I'm using python 3.6 and use Word2vec. I conducted cosine similarity of words using Word2vec and It's working ...
1
vote
1answer
49 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
1
vote
0answers
17 views

Dealing with small number of examples in hierarchical text classification

I am working on a multi-class text classification problem with hierarchical classes structure: super class and sub class for every text example. What am i trying to do is: based on the text predict ...
2
votes
2answers
32 views

How can I do web scrapping of all the articles containing certain keywords mentioned in a dictionary, using Python?

I would like to pull out the title and year of the all articles published in BBC since 2010 based on some keyword search for instance [MBA, LBS, Harvard...... and so on]. How can I do this using ...
1
vote
0answers
9 views

TextRank algorithm for Web content

I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags ...
0
votes
0answers
24 views

Keras model input and output value

I have a dataframe of 2 columns, both text - one is title and other is the label to it. Unique label count is around 40k so one hot encode was out of question. I used word2vec with size=150 for both, ...
0
votes
0answers
14 views

Description matching between two columns

I have a DataFrame with two columns, let's call them column A and column B. Each column is identified with a unique ID and a description. Usually, each idA is in a 1-1 relation with idB. ...
1
vote
0answers
18 views

output of k-mean cluster as collection of tweets

I want to cluster some 1000 tweets using k-mean algorithm. But I don't want the correct output, I just want clustering of tweets. Suppose 1 cluster contain 300 tweets than all the contain of 300 ...
0
votes
0answers
14 views

How to assign label to un-labeld text documents

I have a bunch of text documents and I would like to assign one or more than one label to each document. I know that autoencoders can be used in a semi-supervised setting to first cluster the ...
0
votes
0answers
20 views
0
votes
0answers
84 views

hashingvectorizer vs countvectorizer

I am getting significant performance difference in classifying (junk/non-junk) a large corpus of web-scraped documents when using countvectorizer (CV) vs hashvectorizer (HV) (Overall validation ...
0
votes
0answers
27 views

Intiative detection problem

I currently try to wrap my head around a problem of detecting an initiative in a company report document. The following are the types of initiatives a company can report: health, no_poverty, ...
1
vote
0answers
43 views

What are the rules when extracting SVO triples from preprocessed text?

If you have some already preprocessed text that is tagged, what are the rules to extract SVO triplets if you want a triple like (word, word, word). Can you give the sentence as example and extract all ...
1
vote
1answer
39 views

Help with reusing glove word embedding pretrained model

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?
2
votes
1answer
69 views

Find matching text from a text column

This is my first time to use Data Analytics tool to figure out a solution to a problem. I have a table with following columns ...
0
votes
0answers
99 views

Extract variable names of formula from a document

How can I extract all of the variable names from a given research paper? For example Some applications of algebraic curves to computational vision For example, below is the text and I need to ...
1
vote
0answers
18 views

Common deep learning practices in NLP for text classification

Are there any articles about best modern techniques in text classification (not only for English texts, but in general)? In particular, I'm interested: what kind of text preprocessing techniques are ...
1
vote
1answer
67 views

How to use build_analyzer in sklearn feature extraction

I'm trying to get list of n-gram tokens for text Ex: 'How to use build_analyzer in sklearn feature extraction ' output :['How', 'use', 'build_analyzer', 'sklearn', 'feature', 'extraction', 'How use'...
2
votes
1answer
35 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...
2
votes
0answers
108 views

Alternate of TF-IDF

I had used TF-IDF for text similarity but the results were not so good. I tried to implement google universal encoding (tensorflow hub). The results were satisfactory but not upto the mark. Is there ...
1
vote
0answers
42 views

Proximity distance between two strings

How can I calculate the proximity between two strings Ex: I need to find all the names in unstructured text and also need to look if there is any given string (i.e 'title') nearby Input: '...
0
votes
0answers
10 views

Find top 3 themes or tags for a piece of user supplied text

I am using Multinomial Naïve Bayes to tag a piece of text to a theme, based on a training data set. I am following the example here: https://towardsdatascience.com/multi-class-text-classification-with-...
0
votes
1answer
36 views

How to auto tag texts

Suppose we have predefined list of tags Tag #1, Tag #2, ..., Tag #N and we want to assign ...
2
votes
0answers
12 views

Learning to parse elements from text and assigning them to categories

I am trying to generate a table with values parsed from unstructured text. Below are a couple of examples of possibly thousands of entries. For each entry, I would like to identify the title, assign ...
2
votes
1answer
68 views

How to give names/labels to topics in LDA

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
4
votes
1answer
2k views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...
0
votes
0answers
14 views

How to extract event from news article and merge similar event together

TL;DR Is there a way to extract events from news and cluster similar news event together? I am following the ACE methodology currently, is that a viable way? Longer version: I am currently working ...
0
votes
1answer
79 views

How to interpret Hashingvectorizer representation?

I cannot really understand the logic behind Hashingvectorizer for text feature extraction. I can follow the logic of Bag of Word or TFiDF where the features are values for all/certain words/N-grams ...
0
votes
1answer
218 views

Text Similarities: which nlp methods to use?

I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text. Question 1: Which NLP method should I start with? I have tried ...
0
votes
1answer
312 views

Cosine similarity between two folders (1 and 2) with documents, and find the most relevant set of documents (in folder 2) for each doc (in folder 2)

I have one folder named iir, it has 500 txt files. I have another json file named video (with dictionary structure). I wish to compute: for each of the 500 txt files, find the cosine similarity with ...
1
vote
1answer
56 views

Extracting sections from document based on list of keywords - Python

I am new to NLP and I would like to ask how can I extract sentences from the text based on keywords that I have using Python. I created a list of keywords which will be used to extract sentences from ...
0
votes
2answers
141 views

How to select features for Text classification problem

I am working on a problem where we need to classify user query into multiple classes. Problem: Suppose we are running a website for selling products. The ...
1
vote
0answers
22 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
0
votes
1answer
23 views

How to implement a basic query management and recommendation system

I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see ...
0
votes
0answers
12 views

Expanding entity extraction to certain types of named-noun phrases (Bi-grams or tri-grams)

I have a corpus of articles from business publications. I'm interesting in extraction information relating to specific types of technologies and processes, as well as just interesting phrases. ...
0
votes
1answer
42 views

Extracting specific portions of text from poorly formatted document?

I have a corpus of text files (free books) which are poorly formatted. The goal is to extract a particular chapter (say chapter 2) from the raw text with all weird formatting removed. Some documents ...
0
votes
2answers
152 views

How to detect phrases from an English sentence.

The question is not about detecting keyphrases. It is about detecting a combination of words makes a valid phrase or not. For example, "John reads New York Times in New York." Here, the phrases ...