Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

Extracting Names using NER | Spacy

I'm new to NER and I've been trying to extract names using Spacy. Here's my code: ...
1
vote
1answer
13 views

How many words should be taken as features in a ML problem?

I would like to ask you how many words should be taken as features in a ML program. For example, if I have 30000 distinct words to make a vocabulary, what would a good number be? I am currently ...
0
votes
0answers
15 views

Extracting structured data from semi structured data

I want to use machine learning and NLP to convert semi-structured data in text files to structured data by predicting the patterns in the files and splitting the fields for example if I have a text ...
-1
votes
0answers
27 views

Is there a way to duplicate time series data?

Context: I want to work with a corpus of ~800,000 scientific abstracts from 1970-2017 and predict trends in another corpus. Problem: There may be insufficient data to accurately predict trends between ...
0
votes
0answers
14 views

Analysis of table to make compressing model

I have a table with hundred of entries in this style: //HEADER//CODE1//decimals_here//CODE4//decimals_here//CODE3//decimals_here//CODE5//decimals_here//ENDING// I ...
2
votes
1answer
13 views

Extracting events with attributes from unstructured text

I am scraping websites of organisations (mostly retailers) and I want to use NLP to extract information from the websites’ unstructured text. The first thing I want to do is to identify covid-related ...
1
vote
0answers
36 views

Learning words embedding for bigrams and unigrams in a corpus

I am working on a topic modeling for tweets projects. I have generated my topics using both unigrams and bigrams. Topics are defined with a mixture of both bigrams and unigrams. Now I am planning to ...
2
votes
1answer
47 views

How to evaluate the quality of speech-to-text data without access to the true labels?

I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription ...
1
vote
1answer
27 views

How to find coherence between a large number of sentences

I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
1
vote
1answer
22 views

How to determine whether a semantic concept is present in a string

I need to find a way to detect if a large string contains a specific substring. Imagine that I have a full contract page converted to string in my Python program. What I want to do is to say if a ...
1
vote
0answers
19 views

Create a weighted graph from a continous text

My goal: plot a graph that shows how much the words of a large text are connected to each other. My idea is that the closer the range between words, somehow affected by the occurence, the more ...
1
vote
1answer
15 views

Which phrase should be returned in case of multiple matches when comparing text?

I want to compare one sentence to some other sentences using the Bag of Words model. Suppose that my comparing sentence is: I am playing football and there are three more sentences that I want to ...
1
vote
1answer
19 views

Matching Data Text of Two Place with Exception [closed]

I have data of two places name and it's address in a row and i have to match it. Data is text type, I have read, it have to convert to numeric type that generated by the text. I extracted the numeric ...
3
votes
1answer
45 views

NLP: Information extraction

I need to extract product names from a text column in a dataset. Currently I'm using regular expressions to extract the product names from the middle of the text, but sometimes the product name is ...
0
votes
2answers
42 views

Python library to detect a bank/financial institution name in a string

I would like to extract bank names from a given text like wells Fargo, chase....is there a python library for this? I know there is entity tagger in space and flair but they only identify the entity (...
1
vote
1answer
18 views

String similarity algorithms for string containment (rather than string equality)

I saw that there are a lot of string similarity algorithms to whether two strings are the same. I have a slightly different problem - I get two strings "a" and "b" and I need a ...
0
votes
1answer
37 views

Unsupervised text classification with R/Python

I am relativity new to machine/deep learning and NLP. As a part of my Phd thesis I have scraped vast number of job vacancies (most of them are in Polish, and about 10% are in English ones) and then ...
0
votes
0answers
23 views

How to add time as a feature into word embeddings?

I have a text corpus and I'm using TfidfVectorizer. Would it be possible to cluster the resultant matrix once I concatenate tf-idf vector and time feature matrix I built([year, month, day])? I'm also ...
1
vote
1answer
18 views

Quality check for preprocessing of Text data

I have developed a pipeline for text data preprocessing with different clean up techniques like Stemming , Lemmatization, Stop words removal etc. But now the ask from the business team is to quantify ...
0
votes
0answers
21 views

Convert TFIDF Values to Vector Space Model

I'm working on a project using tf-idf values and cosine similarity for clustering. As my database (elasticsearch) provides tfidf values out of the box (term_freq & doc_freq), my code involves ...
0
votes
0answers
9 views

What does MiniBatchKMeans fit's reassignment_ratio parameter do exactly?

I am using scikit-learn MiniBatchKMeans to do text clustering. In the constructor method, there is a parameter reassignment_ratio, which is described in the ...
0
votes
1answer
35 views

How to interpret the sample_weight parameter in MiniBatchKMeans?

I am using scikit-learn MiniBatchKMeans to do text clustering. In the fit() function there is a parameter sample_weight described as follows: The weights for each ...
1
vote
1answer
34 views

Comparing TFIDF vectors of different shapes

I'm working on a project using TF-IDF vectors and agglomerative clustering -- the idea is that the corpus of documents increases over time, and when a new document is added, the mean cosine similarity ...
0
votes
0answers
14 views

LDA-like algorithm but at the character level?

I have a catalog of products and I'd like to find "topics" in their description. The problem is that you might find in the description things like GraphicsCardVendor10.2 ...
1
vote
0answers
9 views

NER_Multiple_entities

I am working on a problem of entity extraction which requires me to extract variables of interest from a text document. My challenge is that the text contains multiple entities of a variable, for ex. ...
2
votes
1answer
38 views

Correlation among features (e.g. doc length, punctuation, … ) in classifying spam emails

I extracted some other features from my dataset regarding punctuation, capital letters, upper case words. I got these value: looking at the correlation with my target variable (1=spam, 0=not spam), ...
0
votes
1answer
32 views

Text mining in Amazon product review using R. I wasn't able to extract the particular product's review

Text mining on Amazon product review using R Program. I wasn't able to extract the particular product's review(i.e.If iphone 11 has 6k review, I need to extract all of it.) I'm getting only one column ...
1
vote
0answers
22 views

Text Analysis : Recommendation to identify cause of loss from claim narrative documents

I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify ...
1
vote
0answers
27 views

How can I make a better unsupervised text classifier model? Is POS tagging part of Machine Learning and Data science?

I have got complaints data, which is not good, and often contains less than 3 words in every complaint (sometimes so short that only one word of them makes sense). The objective is to find what's ...
2
votes
1answer
33 views

Feature Selection - Conditional Entropy

I've developed an algorithm to define conditional entropy for feature selection in text classification. I'm following the formula at Machine Learning from Text by Charu C. Aggarwal (5.2.2). The author ...
0
votes
0answers
15 views

Classify Spanish Text into different Categories

I want to recommend articles to users depending upon what type of article is user reading, Music, Movies, Politics, etc. I have 3 features: Page Title, Labels, article content. I am using an API (...
1
vote
1answer
25 views

Construct word2vec (CBOW) training data from beginning of sentence

When constructing training data for CBOW, Mikolov et al. suggest using the word from the center of a context window. What is the "best" approach to capturing words at the beginning/end of a ...
0
votes
1answer
30 views

Inference from text data without label or Target

I have a use case where I have text data entered by an approver while approving of some loan. I have to make some inferences as to what could be the reasons for approval using NLP. How should I go ...
3
votes
0answers
275 views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
0
votes
0answers
7 views

Can I use LSI (Latent Semantic indexing) to get similar docs for several documents at the same time?

I'm working on a Recommander system in which I'm using LSI to get similarities between videos. I wonder if I can provide to LSI matrix more than one document and get similar docs for all those. In the ...
1
vote
1answer
39 views

Creating a valid dataset for obtaining results

I have created a domain-specific dataset, lets say it is relating to python programming topic posts. I have taken data from various places specific to this topic to create positive examples in my ...
0
votes
1answer
116 views

Extract data from facebook

I am learning about social media analysis. I am aware that we can extract the data from twitter using hashtags and API. Ex; If I use #covid19, I will get all tweets ...
2
votes
2answers
75 views

Learn (common) grammar / pattern from set of sample strings?

So I currently have a text pattern detection challenge to solve at work. I am trying to make an outlier detection algorithm for a database, for string columns. For example let's say I have the ...
2
votes
1answer
60 views

is it wrong to use average='weighted' when having only 2 classes?

In the book 'Text Analytics with Python', the author provides model_evaluation_utils.py In the code of the .py he does: ...
0
votes
1answer
23 views

how come accuracy_score recognizes the positive label and precision_score does not?

I am executing this code which works perfectly for me: (I only have 'positive' and 'negative' sentiments): ...
0
votes
0answers
25 views

Algorithm for rectifying incorrect text from OCR

I have an OCR (tesseract) engine running on processing "chyron" text from a live stream news feed. However, the OCR is often inaccurate and ends up with some noisy words: ...
3
votes
1answer
2k views

How to create ROC - AUC curves for multi class text classification problem in Python

I am working on a multiclass text classification problem and trying to plot ROC Curve but no success so far. Tried many solutions available but didn't work. Kindly please someone help me out with the ...
2
votes
2answers
35 views

Build a sentiment model from scratch

I would like to know how I can create a sentiment model from scratch. I have my data, list of texts, with no labels about sentiment. ...
3
votes
1answer
265 views

Decision tree and SVM for text classification - theory

I used 4 classifiers for my text data: NB, kNN, DT and SVM. As for NB and kNN I fully understand how they work with text - how we can count probabilities for all words in NB and how to use similarity ...
2
votes
1answer
33 views

Classify text as logical/ not logical

Can some one advise me direction where to look in.Or some resources. Here is a task: User leaves feed back-text with min 50 characters. I need to check if it's normal human sentences/ word ...
1
vote
1answer
74 views

Extracting text into columns

I have a set of customer reports, each is in ms word file. they are all in a similar pattern, for example they start with Name: --, Age: --, Date: --, etc... is there a way to extract particular ...
0
votes
1answer
33 views

How to identify similar words as input from a dictionary

Let's say I have a CSV file (single column) with a list of words as input. Meaning the file looks like below Input Terms ...
1
vote
1answer
38 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
3
votes
3answers
84 views

Better approach to assign values to determine potential fake sentences

I am trying to assign different values for each sentences based on information about the presence of hashtags, upper case letters/words (e.g. HATE) and some others. I created a data frame which ...
0
votes
0answers
11 views

Multi-documents text annotation tool?

I have been looking for a good annotation tool/interface for text annotation. My requirements are as follows: Display of multiple short-text social media posts on the same page, as one task; ...

1
2 3 4 5
11