Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1
vote
2answers
730 views

Which libraries in Python are there in NLP to tokenize the Hindi sentence?

For English language there are libraries like NLTK, CoreNLP which are used for Text Normalization, Word Tokenization and Detokenization, Sentence Splitting etc. Like English, is there any library to ...
2
votes
0answers
37 views

Assigning tags to posts using predefined set of tags

I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. ...
1
vote
1answer
32 views

What's the fastest way to do a text analysis over user reviews on a website for a beginner? [closed]

I want to analyse user reviews for certain products as part of a research project without having to learn analytics from scratch, as my requirement is temporary. I need to do the following: The user ...
0
votes
1answer
54 views

What is the best approach to perform information extraction from tourist reviews using NLP, DL?

I am interested in performing some information extraction from tourist reviews about different places. I have data of 50 different places and around 300-400 reviews about each of them and I would ...
1
vote
1answer
223 views

Does it make sense to use TF-IDF to extract most important tokens from a corpus?

I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus. My understanding of TF-IDF is that it is calculated per token per document, so ...
-1
votes
1answer
19 views

Identifying specific words in text

Let's say I have the following text" Is that another kitten playing in the shoes in the top right? I would like my code to extract kitten from that text. Is ...
2
votes
0answers
62 views

Text Mining with Pubmed Widget Orange

When I was running the text mining I did not have an issue for 57 different searches. I was able to retrieve all of the records regardless of how many there were. Until these 2 errors popped up. I ...
1
vote
0answers
114 views

Term Co-occurence Matrix

I am doing text mining with Orange. How can I use my bag of words to create a term co-occurrence matrix? I am hoping to use this to plot some cool bigram and semantic networks.
0
votes
1answer
36 views

Classify documents using a set of known vocabularies

I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents). One way I am thinking about is to go online ...
2
votes
1answer
57 views

Comparing files (text sources) in Orange

What is the best method to compare text files in Orange? How do I find the difference or similar between two texts and relations between main nodes?
1
vote
0answers
62 views

Use sentiment analysis on word level (aspect level)

I am trying to use Orange 3.20 Text Mining nodes to classify sentiments using a lexicon approach (either using the Method from Liu Hu or Vader). After selecting the columns, I can see the documents ...
3
votes
4answers
79 views

Best way to combine two similar document

I have f.ex.: two news-articles that report the same event. However, these two text are similar BUT not the same. I would like to combine these two texts creating one text that contains only the most "...
1
vote
1answer
116 views

How to create corpus file?

I have text data in .xlsx or .txt files. How do I convert them to a corpus file (.tab)? I would like to use the corpus file with Orange's text add-on.
1
vote
0answers
64 views

R: Using Stanford Corenlp to extract sentence from original text

I am creating a summarizer in R of news articles. Hence, I am building an algorithm to get total score of each sentence based on sum of frequency of lemma in the sentences of the article, annotated ...
2
votes
0answers
28 views

identifying the primary and secondary keywords in sentense

want to identify the primary and secondary keywords which are having an impact to sentences or comparison between 2 keywords. below is the example India and China has highest population in the ...
5
votes
1answer
68 views

Text extraction from large pool of documents of different formats

I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, ...
2
votes
2answers
67 views

Need help with entity tagging

I need to design a system which can identify movie and production company names in a sentence. The approach that comes to my ...
-1
votes
1answer
35 views

How can I categoriese / classify a cluster of words?

I am just wondering if it is possible to classify word clusters? For example if I provide you an array of words [bird,chicken,dock,park,apple,grapes,furits,juice] ...
1
vote
0answers
45 views

Is it a good idea to train Neural Network for classification on dataset where each document has a different class i.e. no class is repeated again?

My goal is to build a recommendation model for which I want to use Neural Network (LSTM). The user will give some input keywords and the model should return the suggestions (classes) based on ...
3
votes
3answers
5k views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
1
vote
1answer
108 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
1
vote
1answer
699 views

What are CRF (Conditional Random Field)

Looking for language modeling, I have been finding CRF in a lot of places which is but looking online for the same isn't actually helping me a lot. I referred Edwin Chen's blog and Ravish Chawala's ...
0
votes
1answer
63 views

Comparing English word pronunciation complexity

I'm trying to figure out a way to compute a score for the pronunciation of a given english word, so I can use that score to compare the pronunciation complexity between english words. Eg: Given ...
1
vote
0answers
105 views

NLP to recognize the meaning of a paragraph

How can I apply NLP to extract the summary of a paragraph,I found this but was wondering if there is a better and easy way before implementing it, as the ans seems to be an old one. Basically what I ...
1
vote
0answers
40 views

Anomaly detection in structured textual data

Pls refer screenshot for sample data. As can be seen most of the fields in data are textual and highly correlated but each row has unique values and hence won't be right to call it categorical. I ...
2
votes
0answers
389 views

Hierarchical classification with multi-class predictor for every parent node

Edit: It turned out that I had an error in my function to compute the combined probabilities (a typo that changed the behavior of my function quite a bit without giving me an error message). Without ...
1
vote
2answers
936 views

extraction information from resume

I have a project in machine learning in which I need to analyze a curriculum vitae. for that I have to write a python program. It uses basic techniques of Natural Language Processing like word ...
5
votes
3answers
199 views

How does Doc2Vec treat numerical data which is a part of text data?

I have data containing both numbers and raw text related differently like: The power of diesel generator is 15kva. I need a single phase generator. Three phase generator required of 140 kva. Need 70g/...
7
votes
2answers
3k views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
1
vote
0answers
73 views

Extracting text from few areas on product label

I'm trying to achieve algorithm that will extract text from few areas(marked with red color) on label(similar to attached image) and QR code on a single photo taken with mobile camera so label may be ...
1
vote
0answers
502 views

Installing NLTK using WHL file -

I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please? https://pypi.org/project/nltk/ The problem ...
1
vote
0answers
13 views

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python. If I do topic modelling and get different groups of topic, how do I pick out the only topic ...
1
vote
0answers
24 views

How to read in all text files from UNIX bash directory in Cloudera's Python API

I'm still pretty new to Cloudera and using the UNIX environment. I have written a mapper that reads in .txt files from a directory in my Windows system, which works just fine. I read files in like ...
1
vote
1answer
144 views

Document parsing modeling and approach?

I'm relatively new to data science / machine learning (yes, I know) and am experimenting with text analysis. I only want a relatively naive approach and am looking to know whether my approach is valid ...
1
vote
0answers
95 views

How do I get Text Mining on Orange version 3.18 [closed]

I have downloaded version 3.18 but there are no Data options for Text Mining (i.e. Corpus Viewer, Word count). Is there a later version of Orange I should download or is there a specific Add On I ...
2
votes
2answers
2k views

Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity. For example : string one : 'Pair of women's ...
1
vote
0answers
71 views

What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content [closed]

Think insurance/medical forms that come from different companies. There is no standard on formatting. I am trying to extract the text based on each section of a given form. A form might have a ...
0
votes
2answers
256 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
2
votes
1answer
34 views

Efficient way to search list of items in a text document

I have a list of items (size ~50K) and several documents( average page per document ~10). I am trying to find what all items are listed in each document as follows : ...
1
vote
1answer
252 views

Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the ...
1
vote
1answer
952 views

Word embeddings for Information Retrieval - Document search?

What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a ...
1
vote
1answer
92 views

Guidelines for vocabulary sizes for BoW

I am currently trying to get a vocabulary for BoW-vector generation out of a set of 200k scientific abstracts. I do some basic filtering of tokens already like lowercasing, stop-word-removal, ...
1
vote
0answers
65 views

Doc2Vec for dataset with several text fields: concatenate or separate models?

I have a dataset with several fields: description, name, header. I want to train doc2vec out of it, so that I could use vectors for classification. So I wonder, ...
1
vote
0answers
104 views

Constructing a graph where any node can have the highest PageRank among all nodes

I'm trying to solve q5.1.4 in mmds chapter5, however, I'm not sure how I can even start. The question is: Construct, for any integer n, a Web such that, depending on β, any of the n nodes can have ...
2
votes
1answer
34 views

Predict votes of future comments [closed]

I have a database with a lot of comments, each comment has a vote, a vote can be positive or negative. ex : -2, -5, -90, +45, +20... So based on this training dataset I want to predict votes of ...
0
votes
1answer
32 views

density of a synset

I am reading the paper Text Classification Using WordNet Hypernyms. In it, the author gives the definition of synset density as the number of occurrences of a synset in the WordNet output divided by ...
1
vote
1answer
49 views

Recurrent Neural Networks Over Multiple Documents Over Time

So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on ...
0
votes
1answer
1k views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...
1
vote
2answers
312 views

What is the best way to use word2vec for bilingual text similarity?

I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this: ...
2
votes
1answer
44 views

Grouping domain specific words/phrases with same meaning

I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/...

1 2
3
4 5
11