Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
1answer
31 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
1
vote
1answer
154 views

What are CRF (Conditional Random Field)

Looking for language modeling, I have been finding CRF in a lot of places which is but looking online for the same isn't actually helping me a lot. I referred Edwin Chen's blog and Ravish Chawala's ...
0
votes
1answer
30 views

Comparing English word pronunciation complexity

I'm trying to figure out a way to compute a score for the pronunciation of a given english word, so I can use that score to compare the pronunciation complexity between english words. Eg: Given ...
0
votes
0answers
60 views

Multi-class string classification

Currently working on Resume Rarser tool using doc2vec. The main assumption that I take when parsing resume is that each line of text (docx, pdf etc) contains information of one class. Although ...
1
vote
0answers
43 views

NLP to recognize the meaning of a paragraph

How can I apply NLP to extract the summary of a paragraph,I found this but was wondering if there is a better and easy way before implementing it, as the ans seems to be an old one. Basically what I ...
1
vote
0answers
27 views

Anomaly detection in structured textual data

Pls refer screenshot for sample data. As can be seen most of the fields in data are textual and highly correlated but each row has unique values and hence won't be right to call it categorical. I ...
2
votes
0answers
147 views

Hierarchical classification with multi-class predictor for every parent node

Edit: It turned out that I had an error in my function to compute the combined probabilities (a typo that changed the behavior of my function quite a bit without giving me an error message). Without ...
1
vote
2answers
377 views

extraction information from resume

I have a project in machine learning in which I need to analyze a curriculum vitae. for that I have to write a python program. It uses basic techniques of Natural Language Processing like word ...
5
votes
3answers
148 views

How does Doc2Vec treat numerical data which is a part of text data?

I have data containing both numbers and raw text related differently like: The power of diesel generator is 15kva. I need a single phase generator. Three phase generator required of 140 kva. Need 70g/...
4
votes
2answers
610 views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
1
vote
0answers
40 views

Extracting text from few areas on product label

I'm trying to achieve algorithm that will extract text from few areas(marked with red color) on label(similar to attached image) and QR code on a single photo taken with mobile camera so label may be ...
1
vote
0answers
192 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
13 views

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python. If I do topic modelling and get different groups of topic, how do I pick out the only topic ...
1
vote
0answers
23 views

How to read in all text files from UNIX bash directory in Cloudera's Python API

I'm still pretty new to Cloudera and using the UNIX environment. I have written a mapper that reads in .txt files from a directory in my Windows system, which works just fine. I read files in like ...
1
vote
1answer
84 views

Document parsing modeling and approach?

I'm relatively new to data science / machine learning (yes, I know) and am experimenting with text analysis. I only want a relatively naive approach and am looking to know whether my approach is valid ...
1
vote
0answers
66 views

How do I get Text Mining on Orange version 3.18 [closed]

I have downloaded version 3.18 but there are no Data options for Text Mining (i.e. Corpus Viewer, Word count). Is there a later version of Orange I should download or is there a specific Add On I ...
2
votes
2answers
876 views

Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity. For example : string one : 'Pair of women's ...
1
vote
0answers
49 views

What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content [closed]

Think insurance/medical forms that come from different companies. There is no standard on formatting. I am trying to extract the text based on each section of a given form. A form might have a ...
0
votes
1answer
52 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
0
votes
0answers
104 views

extract calendar event information from unstructured text

I am trying to automate event creation from email. Below is an example, I cannot make any assumptions about the format. Consider an email message like below. ...
2
votes
1answer
27 views

Efficient way to search list of items in a text document

I have a list of items (size ~50K) and several documents( average page per document ~10). I am trying to find what all items are listed in each document as follows : ...
0
votes
0answers
14 views

Verify compliance with the laws in text documents

USE CASE - AS IS A user sends a text document X to a system where it requires a s service (for example, to request a residence permit). A technician examines the document verifying that the document ...
1
vote
1answer
140 views

Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the ...
1
vote
1answer
482 views

Word embeddings for Information Retrieval - Document search?

What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a ...
1
vote
1answer
43 views

Guidelines for vocabulary sizes for BoW

I am currently trying to get a vocabulary for BoW-vector generation out of a set of 200k scientific abstracts. I do some basic filtering of tokens already like lowercasing, stop-word-removal, ...
1
vote
0answers
44 views

Doc2Vec for dataset with several text fields: concatenate or separate models?

I have a dataset with several fields: description, name, header. I want to train doc2vec out of it, so that I could use vectors for classification. So I wonder, ...
0
votes
0answers
14 views

Spatially encoding textual data

Imagine we have a textual data that must be passed to convolutional neural network. Generally we know that for natural language processing, textual data is embedded into vector space, but in this ...
0
votes
0answers
31 views

I want to extract the problem behind the question using NLP/NLU

I want to convert a given question to the problem behind it. For instance, This question "What free software can convert audio files into text files ?" hides a problem/action which is "Convert ...
1
vote
0answers
42 views

Constructing a graph where any node can have the highest PageRank among all nodes

I'm trying to solve q5.1.4 in mmds chapter5, however, I'm not sure how I can even start. The question is: Construct, for any integer n, a Web such that, depending on β, any of the n nodes can have ...
2
votes
1answer
29 views

Predict votes of future comments

I have a database with a lot of comments, each comment has a vote, a vote can be positive or negative. ex : -2, -5, -90, +45, +20... So based on this training dataset I want to predict votes of ...
0
votes
1answer
27 views

density of a synset

I am reading the paper Text Classification Using WordNet Hypernyms. In it, the author gives the definition of synset density as the number of occurrences of a synset in the WordNet output divided by ...
0
votes
1answer
19 views

Recurrent Neural Networks Over Multiple Documents Over Time

So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on ...
0
votes
0answers
112 views

Feeding machine learning model with different matrix

Well my question is a general question. I tried to find some relevant information before posting my question here, but no success!. I am working on ...
0
votes
1answer
646 views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...
0
votes
1answer
117 views

What is the best way to use word2vec for bilingual text similarity?

I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this: ...
1
vote
1answer
30 views

Grouping domain specific words/phrases with same meaning

I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/...
1
vote
1answer
29 views

For text classification that has innumerable features, how do I choose the number of neurons and layers for MLPClassifier?

In my use case of text classification (identify the author from a subset of 10 authors), I find that post all processing with trigrams, there are a 100 thousand and odd features with nearly 50k ...
0
votes
1answer
22 views

Composing phrases into a grammatically correct sentence?

I'm wondering if there exist any models which could take in an ordered list of phrases without punctuation and generate a grammatically correct sentence from it. For example, for the input: ["My dog"...
2
votes
3answers
93 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
1
vote
1answer
173 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
1
vote
0answers
28 views

Dealing with small number of examples in hierarchical text classification

I am working on a multi-class text classification problem with hierarchical classes structure: super class and sub class for every text example. What am i trying to do is: based on the text predict ...
2
votes
2answers
73 views

How can I do web scrapping of all the articles containing certain keywords mentioned in a dictionary, using Python?

I would like to pull out the title and year of the all articles published in BBC since 2010 based on some keyword search for instance [MBA, LBS, Harvard...... and so on]. How can I do this using ...
1
vote
0answers
19 views

TextRank algorithm for Web content

I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags ...
1
vote
0answers
32 views

output of k-mean cluster as collection of tweets

I want to cluster some 1000 tweets using k-mean algorithm. But I don't want the correct output, I just want clustering of tweets. Suppose 1 cluster contain 300 tweets than all the contain of 300 ...
3
votes
0answers
145 views

What are the rules when extracting SVO triples from preprocessed text?

If you have some already preprocessed text that is tagged, what are the rules to extract SVO triplets if you want a triple like (word, word, word). Can you give the sentence as example and extract all ...
1
vote
1answer
125 views

Help with reusing glove word embedding pretrained model

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?
3
votes
1answer
279 views

Find matching text from a text column

This is my first time to use Data Analytics tool to figure out a solution to a problem. I have a table with following columns ...
1
vote
0answers
27 views

Common deep learning practices in NLP for text classification

Are there any articles about best modern techniques in text classification (not only for English texts, but in general)? In particular, I'm interested: what kind of text preprocessing techniques are ...
2
votes
1answer
380 views

How to use build_analyzer in sklearn feature extraction

I'm trying to get list of n-gram tokens for text Ex: 'How to use build_analyzer in sklearn feature extraction ' output :['How', 'use', 'build_analyzer', 'sklearn', 'feature', 'extraction', 'How use'...
2
votes
1answer
139 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...