Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

0
votes
0answers
4 views

Clustering text post vectorization by Universal sentence encoder

I have 1000 of datasets within which clustering needs to happen in each of them to find the textual categories. I have vectorized the data using google sentence encoder. What can be the best way to ...
1
vote
0answers
26 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
9 views

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python. If I do topic modelling and get different groups of topic, how do I pick out the only topic ...
1
vote
0answers
21 views

How to read in all text files from UNIX bash directory in Cloudera's Python API

I'm still pretty new to Cloudera and using the UNIX environment. I have written a mapper that reads in .txt files from a directory in my Windows system, which works just fine. I read files in like ...
1
vote
1answer
24 views

Document parsing modeling and approach?

I'm relatively new to data science / machine learning (yes, I know) and am experimenting with text analysis. I only want a relatively naive approach and am looking to know whether my approach is valid ...
1
vote
0answers
22 views

How do I get Text Mining on Orange version 3.18 [closed]

I have downloaded version 3.18 but there are no Data options for Text Mining (i.e. Corpus Viewer, Word count). Is there a later version of Orange I should download or is there a specific Add On I ...
2
votes
1answer
23 views

Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity. For example : string one : 'Pair of women's ...
1
vote
0answers
18 views

What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content [closed]

Think insurance/medical forms that come from different companies. There is no standard on formatting. I am trying to extract the text based on each section of a given form. A form might have a ...
0
votes
0answers
8 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
0
votes
0answers
46 views

extract calendar event information from unstructured text

I am trying to automate event creation from email. Below is an example, I cannot make any assumptions about the format. Consider an email message like below. ...
2
votes
1answer
22 views

Efficient way to search list of items in a text document

I have a list of items (size ~50K) and several documents( average page per document ~10). I am trying to find what all items are listed in each document as follows : ...
0
votes
0answers
11 views

Verify compliance with the laws in text documents

USE CASE - AS IS A user sends a text document X to a system where it requires a s service (for example, to request a residence permit). A technician examines the document verifying that the document ...
1
vote
1answer
13 views

Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the ...
1
vote
1answer
33 views

Word embeddings for Information Retrieval - Document search?

What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a ...
1
vote
1answer
18 views

Guidelines for vocabulary sizes for BoW

I am currently trying to get a vocabulary for BoW-vector generation out of a set of 200k scientific abstracts. I do some basic filtering of tokens already like lowercasing, stop-word-removal, ...
1
vote
0answers
14 views

Doc2Vec for dataset with several text fields: concatenate or separate models?

I have a dataset with several fields: description, name, header. I want to train doc2vec out of it, so that I could use vectors for classification. So I wonder, ...
0
votes
0answers
9 views

Spatially encoding textual data

Imagine we have a textual data that must be passed to convolutional neural network. Generally we know that for natural language processing, textual data is embedded into vector space, but in this ...
0
votes
0answers
21 views

I want to extract the problem behind the question using NLP/NLU

I want to convert a given question to the problem behind it. For instance, This question "What free software can convert audio files into text files ?" hides a problem/action which is "Convert ...
0
votes
0answers
19 views

Constructing a graph where any node can have the highest PageRank among all nodes

I'm trying to solve q5.1.4 in mmds chapter5, however, I'm not sure how I can even start. The question is: Construct, for any integer n, a Web such that, depending on β, any of the n nodes can have ...
2
votes
1answer
22 views

Predict votes of future comments

I have a database with a lot of comments, each comment has a vote, a vote can be positive or negative. ex : -2, -5, -90, +45, +20... So based on this training dataset I want to predict votes of ...
0
votes
0answers
26 views

Text processing on OCR Text

I have a stream of pages that I am looking to perform text analysis (classification, entity extraction, etc.) tasks on. The pages can be semi-structured (forms) and unstructured (letters, etc.) are ...
0
votes
0answers
13 views

Supervised Text Classification (proportions)

Is there any other algorithm (package for R) than the tool kit ReadMe (Hopkins/King) which can give out estimated proportions of categories? The ReadMe software package for R takes as input a set ...
0
votes
1answer
25 views

density of a synset

I am reading the paper Text Classification Using WordNet Hypernyms. In it, the author gives the definition of synset density as the number of occurrences of a synset in the WordNet output divided by ...
0
votes
0answers
29 views

Applying Label From Supervised Learning to Unlabeled Data- Text Classification

I am wondering if anyone has code to following: 1) Apply labels from a previous text classification dataset like this type of data (https://colab.research.google.com/drive/...
0
votes
1answer
13 views

Recurrent Neural Networks Over Multiple Documents Over Time

So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on ...
0
votes
0answers
29 views

Feeding machine learning model with different matrix

Well my question is a general question. I tried to find some relevant information before posting my question here, but no success!. I am working on ...
0
votes
1answer
147 views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...
0
votes
0answers
29 views

What is the best way to use word2vec for bilingual text similarity?

I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this: ...
1
vote
1answer
23 views

Grouping domain specific words/phrases with same meaning

I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/...
1
vote
1answer
19 views

For text classification that has innumerable features, how do I choose the number of neurons and layers for MLPClassifier?

In my use case of text classification (identify the author from a subset of 10 authors), I find that post all processing with trigrams, there are a 100 thousand and odd features with nearly 50k ...
0
votes
1answer
18 views

Composing phrases into a grammatically correct sentence?

I'm wondering if there exist any models which could take in an ordered list of phrases without punctuation and generate a grammatically correct sentence from it. For example, for the input: ["My dog"...
1
vote
2answers
45 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
1
vote
1answer
75 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
1
vote
0answers
22 views

Dealing with small number of examples in hierarchical text classification

I am working on a multi-class text classification problem with hierarchical classes structure: super class and sub class for every text example. What am i trying to do is: based on the text predict ...
2
votes
2answers
45 views

How can I do web scrapping of all the articles containing certain keywords mentioned in a dictionary, using Python?

I would like to pull out the title and year of the all articles published in BBC since 2010 based on some keyword search for instance [MBA, LBS, Harvard...... and so on]. How can I do this using ...
1
vote
0answers
14 views

TextRank algorithm for Web content

I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags ...
1
vote
0answers
23 views

output of k-mean cluster as collection of tweets

I want to cluster some 1000 tweets using k-mean algorithm. But I don't want the correct output, I just want clustering of tweets. Suppose 1 cluster contain 300 tweets than all the contain of 300 ...
1
vote
0answers
72 views

What are the rules when extracting SVO triples from preprocessed text?

If you have some already preprocessed text that is tagged, what are the rules to extract SVO triplets if you want a triple like (word, word, word). Can you give the sentence as example and extract all ...
1
vote
1answer
81 views

Help with reusing glove word embedding pretrained model

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?
3
votes
1answer
139 views

Find matching text from a text column

This is my first time to use Data Analytics tool to figure out a solution to a problem. I have a table with following columns ...
1
vote
0answers
22 views

Common deep learning practices in NLP for text classification

Are there any articles about best modern techniques in text classification (not only for English texts, but in general)? In particular, I'm interested: what kind of text preprocessing techniques are ...
1
vote
1answer
128 views

How to use build_analyzer in sklearn feature extraction

I'm trying to get list of n-gram tokens for text Ex: 'How to use build_analyzer in sklearn feature extraction ' output :['How', 'use', 'build_analyzer', 'sklearn', 'feature', 'extraction', 'How use'...
2
votes
1answer
80 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...
2
votes
0answers
171 views

Alternate of TF-IDF

I had used TF-IDF for text similarity but the results were not so good. I tried to implement google universal encoding (tensorflow hub). The results were satisfactory but not upto the mark. Is there ...
1
vote
0answers
46 views

Proximity distance between two strings

How can I calculate the proximity between two strings Ex: I need to find all the names in unstructured text and also need to look if there is any given string (i.e 'title') nearby Input: '...
0
votes
0answers
12 views

Find top 3 themes or tags for a piece of user supplied text

I am using Multinomial Naïve Bayes to tag a piece of text to a theme, based on a training data set. I am following the example here: https://towardsdatascience.com/multi-class-text-classification-with-...
0
votes
1answer
120 views

How to auto tag texts

Suppose we have predefined list of tags Tag #1, Tag #2, ..., Tag #N and we want to assign ...
2
votes
0answers
14 views

Learning to parse elements from text and assigning them to categories

I am trying to generate a table with values parsed from unstructured text. Below are a couple of examples of possibly thousands of entries. For each entry, I would like to identify the title, assign ...
2
votes
1answer
195 views

How to give names/labels to topics in LDA [duplicate]

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
4
votes
1answer
4k views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...