Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

1
vote
0answers
27 views

Is it a good idea to train Neural Network for classification on dataset where each document has a different class i.e. no class is repeated again?

My goal is to build a recommendation model for which I want to use Neural Network (LSTM). The user will give some input keywords and the model should return the suggestions (classes) based on ...
0
votes
0answers
8 views

Is it possible to calculate precision/recall for a bag-of-words model?

Suppose I have a list of keywords given to a document: {keyword, extract, graph, represent, text, weight, number, document} and then I have the keywords ...
2
votes
3answers
43 views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
0
votes
0answers
10 views

Odd distribution in number of recipients in the ENRON corpus

When plotting the number of mails vs. the number of recipients in a mail from the ENRON dataset, I receive a zig-zag line with the odd numbers counterintuitively always being bigger than the even ...
0
votes
0answers
12 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
0answers
24 views

Multi-label text classification (3 targetted result columns)

I am trying to build a multi-label text classifier for suggesting "AC_location", "Issue" and "Part_Affected" on the generated events. The textual data is full of ...
1
vote
1answer
30 views

What are CRF (Conditional Random Field)

Looking for language modeling, I have been finding CRF in a lot of places which is but looking online for the same isn't actually helping me a lot. I referred Edwin Chen's blog and Ravish Chawala's ...
0
votes
1answer
22 views

Comparing English word pronunciation complexity

I'm trying to figure out a way to compute a score for the pronunciation of a given english word, so I can use that score to compare the pronunciation complexity between english words. Eg: Given ...
0
votes
0answers
10 views

Combiation of stemming and stop word removal consequences on standard errors

I've read an article of Greene, Ceron, Schumacher and Fazekas which called The Nuts and Bolts of Automated Text Analysis: Comparing Different Document Pre-Processing Techniques in FourCountries. In ...
0
votes
0answers
26 views

Multi-class string classification

Currently working on Resume Rarser tool using doc2vec. The main assumption that I take when parsing resume is that each line of text (docx, pdf etc) contains information of one class. Although ...
0
votes
0answers
18 views

Text Mining/Printing with Python

I have a list of users based on frequency of tweets[Here], and a dataset of tweets that looks like [This].I need to write a python script/program that reads my list of users line by line, then for ...
0
votes
0answers
41 views

Multi Class Text Classification?

I have a csv file which contains two columns : Name | Detail Example : French & English code_application | Code application utilisee name_user | first name of user location_place | full ...
1
vote
0answers
36 views

NLP to recognize the meaning of a paragraph

How can I apply NLP to extract the summary of a paragraph,I found this but was wondering if there is a better and easy way before implementing it, as the ans seems to be an old one. Basically what I ...
1
vote
0answers
14 views

Anomaly detection in structured textual data

Pls refer screenshot for sample data. As can be seen most of the fields in data are textual and highly correlated but each row has unique values and hence won't be right to call it categorical. I ...
2
votes
0answers
73 views

Hierarchical classification with multi-class predictor for every parent node

Edit: It turned out that I had an error in my function to compute the combined probabilities (a typo that changed the behavior of my function quite a bit without giving me an error message). Without ...
1
vote
2answers
65 views

extraction information from resume

I have a project in machine learning in which I need to analyze a curriculum vitae. for that I have to write a python program. It uses basic techniques of Natural Language Processing like word ...
0
votes
0answers
12 views

Best text annotation tool for Named Entity disambiguation

I want to disambiguate entites in my corpus. In order to do that I want to annotate them but possible tags for every entity are drawn from different sets. Fake example: Mary went home. Peter has ...
5
votes
3answers
127 views

How does Doc2Vec treat numerical data which is a part of text data?

I have data containing both numbers and raw text related differently like: The power of diesel generator is 15kva. I need a single phase generator. Three phase generator required of 140 kva. Need 70g/...
5
votes
2answers
112 views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
1
vote
0answers
32 views

Extracting text from few areas on product label

I'm trying to achieve algorithm that will extract text from few areas(marked with red color) on label(similar to attached image) and QR code on a single photo taken with mobile camera so label may be ...
1
vote
0answers
69 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
10 views

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python. If I do topic modelling and get different groups of topic, how do I pick out the only topic ...
1
vote
0answers
23 views

How to read in all text files from UNIX bash directory in Cloudera's Python API

I'm still pretty new to Cloudera and using the UNIX environment. I have written a mapper that reads in .txt files from a directory in my Windows system, which works just fine. I read files in like ...
1
vote
1answer
50 views

Document parsing modeling and approach?

I'm relatively new to data science / machine learning (yes, I know) and am experimenting with text analysis. I only want a relatively naive approach and am looking to know whether my approach is valid ...
1
vote
0answers
48 views

How do I get Text Mining on Orange version 3.18 [closed]

I have downloaded version 3.18 but there are no Data options for Text Mining (i.e. Corpus Viewer, Word count). Is there a later version of Orange I should download or is there a specific Add On I ...
2
votes
2answers
232 views

Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity. For example : string one : 'Pair of women's ...
1
vote
0answers
23 views

What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content [closed]

Think insurance/medical forms that come from different companies. There is no standard on formatting. I am trying to extract the text based on each section of a given form. A form might have a ...
0
votes
0answers
15 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
0
votes
0answers
72 views

extract calendar event information from unstructured text

I am trying to automate event creation from email. Below is an example, I cannot make any assumptions about the format. Consider an email message like below. ...
2
votes
1answer
24 views

Efficient way to search list of items in a text document

I have a list of items (size ~50K) and several documents( average page per document ~10). I am trying to find what all items are listed in each document as follows : ...
0
votes
0answers
11 views

Verify compliance with the laws in text documents

USE CASE - AS IS A user sends a text document X to a system where it requires a s service (for example, to request a residence permit). A technician examines the document verifying that the document ...
1
vote
1answer
44 views

Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the ...
1
vote
1answer
166 views

Word embeddings for Information Retrieval - Document search?

What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a ...
1
vote
1answer
21 views

Guidelines for vocabulary sizes for BoW

I am currently trying to get a vocabulary for BoW-vector generation out of a set of 200k scientific abstracts. I do some basic filtering of tokens already like lowercasing, stop-word-removal, ...
1
vote
0answers
28 views

Doc2Vec for dataset with several text fields: concatenate or separate models?

I have a dataset with several fields: description, name, header. I want to train doc2vec out of it, so that I could use vectors for classification. So I wonder, ...
0
votes
0answers
11 views

Spatially encoding textual data

Imagine we have a textual data that must be passed to convolutional neural network. Generally we know that for natural language processing, textual data is embedded into vector space, but in this ...
0
votes
0answers
25 views

I want to extract the problem behind the question using NLP/NLU

I want to convert a given question to the problem behind it. For instance, This question "What free software can convert audio files into text files ?" hides a problem/action which is "Convert ...
1
vote
0answers
28 views

Constructing a graph where any node can have the highest PageRank among all nodes

I'm trying to solve q5.1.4 in mmds chapter5, however, I'm not sure how I can even start. The question is: Construct, for any integer n, a Web such that, depending on β, any of the n nodes can have ...
2
votes
1answer
23 views

Predict votes of future comments

I have a database with a lot of comments, each comment has a vote, a vote can be positive or negative. ex : -2, -5, -90, +45, +20... So based on this training dataset I want to predict votes of ...
0
votes
1answer
25 views

density of a synset

I am reading the paper Text Classification Using WordNet Hypernyms. In it, the author gives the definition of synset density as the number of occurrences of a synset in the WordNet output divided by ...
0
votes
1answer
15 views

Recurrent Neural Networks Over Multiple Documents Over Time

So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on ...
0
votes
0answers
64 views

Feeding machine learning model with different matrix

Well my question is a general question. I tried to find some relevant information before posting my question here, but no success!. I am working on ...
0
votes
1answer
321 views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...
0
votes
1answer
57 views

What is the best way to use word2vec for bilingual text similarity?

I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this: ...
1
vote
1answer
24 views

Grouping domain specific words/phrases with same meaning

I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/...
1
vote
1answer
23 views

For text classification that has innumerable features, how do I choose the number of neurons and layers for MLPClassifier?

In my use case of text classification (identify the author from a subset of 10 authors), I find that post all processing with trigrams, there are a 100 thousand and odd features with nearly 50k ...
0
votes
1answer
18 views

Composing phrases into a grammatically correct sentence?

I'm wondering if there exist any models which could take in an ordered list of phrases without punctuation and generate a grammatically correct sentence from it. For example, for the input: ["My dog"...
2
votes
3answers
61 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
1
vote
1answer
102 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
1
vote
0answers
24 views

Dealing with small number of examples in hierarchical text classification

I am working on a multi-class text classification problem with hierarchical classes structure: super class and sub class for every text example. What am i trying to do is: based on the text predict ...