Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
7 views

Analysis on the 5 segment of the data to differentiate from each other using R

I have 1000 customers operational data which is already divided into segments. Now I have to do analysis on different segments to find out how they differ from each other. Why they fall into their ...
1
vote
2answers
18 views

Visualize/analyze data before or after imputing missing values?

My understanding is that we impute missing values in order to preserve those training examples so our Machine Learning algorithms have as many training examples as possible. To me, it would make ...
0
votes
0answers
7 views

How to perform Learning to Rank for a small dataset

I am very interested in applying Learning to rank to my problem doamin. When I read through the literature of Learning to rank I ...
0
votes
0answers
11 views

Extract data from mainly unstructured sets and derive risk metrics out of those

I have the following question (this was a real life example): Q: Extract data from mainly unstructured sets and derive risk metrics out of those. From what you know or imagine about the data ...
0
votes
0answers
13 views

Class Size Imbalance for LDA or any other Content based analysis

I am running some content analysis studies on my dataset which has two different classes, and each class has a respective list of the document I am analyzing. I compare the LDA topic model inference ...
0
votes
1answer
25 views

Predict Customer Next Purchase with Sequence

Suppose I buy products: [1,2,3,4] Another customer X bought: [2,3] Most probably customer X next purchase will be: 4 Sequence is very important in my problem I tried association analysis using R, ...
1
vote
1answer
20 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
0
votes
1answer
20 views

Extracting information from online job postings

Okay, so I'm trying to build a data set about data science job openings. I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
2
votes
2answers
44 views

Is there a real life meaning about KMeans error?

I am trying to understand the meaning of error in sklearn KMeans. In the context of house pricing prediction, the error linear regression could be considered as the money difference per square foot. ...
2
votes
0answers
22 views

Can we use log transformation for outliers?

I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions, 1) When I skewness and kurtosis for the normality test, two ...
0
votes
0answers
9 views

Is a transaction with 1 item important in apriori?

Im doing association analysis. I have alot of transactions with only 1 item Are they important or should I remove them? because transactions of 1 item doesnt really let you know , if item 1 bought ...
0
votes
0answers
9 views

Identifying patterns/motifs in categorical time series

There is a data set I currently possess of the following form: $D = [(s_0, t_0), (s_1, t_1), ..., (s_N, t_N)]$, where $s_i \in \mathcal{S}, 0 \leq t_i < T$. Here $\mathcal{S}$ is a finite alphabet ...
0
votes
2answers
26 views

Convert non-binary value into binary value?

I'm kind of a newbie in ML I have a dataset like: ...
0
votes
1answer
18 views

Is it possible to build a regression model for predicting movie gross using sections on their wikipedia pages?

I got this as an assignment from a company recruiter and I've successfully scraped a dataset of about 650 movies with their 'Plot', 'Music' and 'Marketing' sections and gross. I've tried tfidf and ...
0
votes
1answer
40 views

Implication of a dominant Principal Component in PCA analysis

I need help, are there any practical implications of a dominant principal component. For example, if of three PCs, PC1 explains almost 100% of the variance in this dataset, What does this mean in ...
0
votes
0answers
24 views

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file

CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar', 'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site- packages\tabula\tabula-1.0....
1
vote
1answer
41 views

Why does np.linalg.eig produce an opposite-signed eigenvector?

I am learning SVD by following this MIT course. In this video, the lecturer is finding the SVD for $$ \begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix}, $$ which involves finding the ...
1
vote
1answer
87 views

is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

I am learning SVD by following this MIT course The lecturer is trying to normalize a vector $${\begin{bmatrix} -3\\ 1\\ \end{bmatrix}}$$ to $${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\...
1
vote
0answers
23 views

Speed of SVM.. Poly kernel method runs endlessly and never completes execution

I'm using scikitlearn in Python to create some models while trying different kernels. I was surprised to see that rbf was fit in under a second, whereas linear took a minute and poly took hours. Can ...
0
votes
1answer
42 views

Why n-split is not possible for a dataframe with KFold?

On running below code on python 3.7, I am getting the following response: 'DataFrame' object has no attribute 'n_splits'. How to get rid of this? ...
0
votes
0answers
21 views

Reading pdf file with all pages in Python error: CalledProcessError:

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file. ...
2
votes
1answer
33 views

How valuable is a categorical feature that has a predominant category over all other ones?

Is a categorical feature that has almost equally distributed in it's category more important or the one which one of it's category is predominant over all other ones? In data prepossessing step for "...
0
votes
1answer
32 views

What are good practices in reporting RMSE or MAPE estimates for a machine learning model?

I have built a regression model to estimate prices given some attributes. Prices are expressed in million USD, with average of 4.4 million USD. I need to provide an estimate of the RMSE and MAPE of ...
3
votes
0answers
83 views

How to apply rolling volume profile function with a pandas TimeDelta

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo. Here the author gets slice the index and apply the main function by ...
-2
votes
2answers
36 views

What is Data Science and Machine Learning and what language mostly used to program? [closed]

I am newbie in Data Science, Machine Learning and any related to data science but I want to try it. Unfortunately, googling makes it tedious and complicated so I hope to be answered by anybody who's ...
1
vote
1answer
30 views

What's the fastest way to do a text analysis over user reviews on a website for a beginner? [closed]

I want to analyse user reviews for certain products as part of a research project without having to learn analytics from scratch, as my requirement is temporary. I need to do the following: The user ...
1
vote
1answer
59 views

Extract text from a image - OCR

This is the first time I am working with OCR. I have an image and want to extract data from the image. My image looks like this: I want to extract the parameters and the values against them. Can ...
2
votes
0answers
81 views

Out of stock / Spike in demand prediction

The goal is to predict out-of-stock situations, either quantitatively (the gap) or qualitatively (out-of-stock likely to happen in next few weeks). Background: We have existing demand planning ...
1
vote
1answer
47 views

How to run KNN (or other) on nested features? Image metadata

I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I ...
1
vote
1answer
20 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
2answers
41 views

Data Extraction from pdf [closed]

I have a requirement where I need build a solution to extract information from multiple pdfs. The information is stored in tabular format in pdf files. The tables are not very well structured and each ...
0
votes
1answer
25 views

How to club the orders in such a way that maximum number of items are common amongst them?

Consider the following data set: The above table shows the quantity of each item used in the orders SO1 SO2 etc. I need to club the orders in such a way that maximum number of items are common amongst ...
0
votes
1answer
31 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
0
votes
1answer
19 views

How can I train a many-to-one RNN with an array of 2D matrices?

I have eye tracking data for every word of a novel. Features for every word is given separately. I want to take groups of 100 words to make a sample and then use each of these samples as a single ...
0
votes
1answer
14 views

Parsing and storing a large amount of HTML data

I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch ...
0
votes
1answer
42 views

Binary Encoding of Ordinal Categories

I have a data frame in which one of my columns is the target value and there are lots of ordinal categories in columns of data frame. I want to encode these ordinal categories in columns with this ...
1
vote
3answers
59 views

Test data significantly different from training data

What are the risks if the test data is significantly different from the training data? Is the most significant problem related to both?
0
votes
0answers
7 views

How to mine spatial telemetry data

I have a dataset of telemetry data that consists of -- Unique ID geohash DTG (yyyy-mm-ddTHH:MM:SS.000UTC) Lat Long and I'm trying to see if there are associations between ID's within the geohash ...
1
vote
1answer
33 views

TDIDT Decision Trees algorithm

What is the Difference between TDIDT, ID3, CART, and C4.5? My main concern is about TDIDT, Is it first ever algorithm that came with Decision trees? Is it predecessor or successor of ID3, CART, ...
2
votes
1answer
33 views

Calculate a ranking function from classification features

I am using 3 features (x1, x2, x3) for binary classification. All my feature values are in 0 to 1 range (unit range). I obtained how important each feature was in ...
1
vote
1answer
23 views

When and where dummy.data.frame would be used?

I am learning this post, in which, the author gives the piece of code: ...
0
votes
0answers
11 views

How to solve a tie in one-r?

when using the one r algorithm with this data a triple tie happens, age group = 10/12, fresh fruit = 10/12 and id = 10/12 why does weka choose age group? Age group: Senior -> Anchovy ...
0
votes
1answer
54 views

Cohort analysis over 3 months

I am trying to look at the customer retention & churn by using cohorts for an e-commerce usecase. From a business perspective, a client is defined as churned if it hasn't performed any ...
0
votes
0answers
13 views

What is the accepted level of persona coverage for Recommender Systems?

Almost all of e-commerce companies use recommender systems which involve a set of personas. (explained in this post). "Your personas will never cover 100% of your users" In practice, what is the ...
1
vote
0answers
38 views

similarity measures for categorical data in data mining

I am working on my assignment in which i have to mention 5 similarity measures for categorical and continuous data in data mining.As a beginner i tried my best and found SQUARE DISTANCE,EUCLIDEAN AND ...
0
votes
0answers
11 views

is it possible to run a Monte Carlo simulation with a single sample? I guess not

I am learning data science through a series of HarvardX courses. to elaborate Sample Correlation, the professor assumes A less fortunate geneticist can only afford to take a random sample of 25 ...
0
votes
0answers
6 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
0
votes
0answers
11 views

Determine learned topics in text

I have a large number of texts (each about 1000 words). Every text contains a various number of topics, usually expressed as a sentence. I've extracted and categorize 22 topics and found 5000 of them ...
0
votes
1answer
29 views

What is the convergence criteria of a semi-supervised learning algorithm?

I would like to know when to stop doing semi supervision? For example, if I learn a classifier from a small dataset and then use it to label a pool of unlabelled dataset. I then use the newly labelled ...