Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
2
votes
3answers
64 views
+50

References for longitudinal data analysis

So my goal is to study longitudinal data (data in time series) by applying some data mining techniques. Ultimately I want to be able to "predict" outcomes. For example, a study of patients along the ...
-3
votes
0answers
13 views

Is Data Mining easier then Theoretical Machine Learning ad Deep Learning [on hold]

Is data mining (text mining for example...) easier than theoretical machine learning and deep learning?
1
vote
1answer
17 views

Infer family type, size from reviews

I have a bunch of reviews: ...
1
vote
0answers
38 views

Dependency Graph

I want to create a dependency graph of some sensors in the network ( based on their reported value). Please note that a change in the values of sensors is related to each other. For example, if the ...
3
votes
0answers
19 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
-1
votes
0answers
10 views

Clustering a dataset using Weka when a categorical class is not available

I'm analyzing the Boston housing dataset using the Weka tool. I need to use algorithms from both classification and clustering tabs. The columns fo the Boston dataset is like below MEDV is the ...
0
votes
0answers
13 views

How to calculate Fuzzy C-Means problem by hand

I figured that this doubt next can interest another students like me and help others also that are trying to understand mathematically the fuzzy c-means mathematical mechanism already that some books ...
-1
votes
0answers
11 views

Find the valid Range of data [closed]

I am researching with IOT where my major is Data Mining. I have a device that collects 5 parameters of data from a Tree plant. Light Intensity Temperature Humidity Soil Moisture Critical Environment ...
1
vote
1answer
20 views

T-DBSCAN - Implementing STOP logic

I am attempting to implement the T-DBSCAN algorithm described in T-DBSCAN: A Spatiotemporal Density Clustering for GPS Trajectory Segmentation. I have been able to implement most of the logic between ...
0
votes
0answers
17 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
2
votes
1answer
40 views

measuring flip-flop behaviour across several topics

I'm trying to analyze a behavior called "sentiment flipping" of users in a dataset, but I'm not able to step on. Let's suppose that I have two groups of users, say them good and bad users. My ...
2
votes
3answers
55 views

Dealing with no data

I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with ...
1
vote
1answer
29 views

Does AWS Sagemaker continue running when I go offline?

I'm pulling data from web services in an AWS Sagemaker Notebook. I'd like to be able to close my computer for an hour, reconnect my computer and see that the data collection has continued to run. ...
1
vote
1answer
21 views

some algorithms to identify of a pattern in text data

I have few sentences like below in my project(around 25000), ...
0
votes
1answer
48 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
0
votes
1answer
12 views

What are some key difference in functionnality between ProM and Apromore?

I would like to know what are some use cases where one would use ProM and where one would use Apromore. I have traces and events from logs that I would like to use process mining on (mapping a ...
0
votes
2answers
22 views

How to train a model on a data where there are multiple data inside a data point?

I'm trying to do prediction on capacity column, however each data point consist of more data. Each data point represent a cycle data. Each cycle has a capacity. Each cycle runs for some time duration,...
2
votes
3answers
700 views

Pandas change value of a column based another column condition

I have values in column1, I have columns in column2. What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90. Here is what i did so ...
0
votes
3answers
33 views

Problem in implementing CNN

I am using the fashion MNIST dataset to try to work this out: I am using the data from the links: Training : http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ...
0
votes
0answers
15 views

Why FP-Growth is slower than Apriori?

My project is about Data mining (in MATLAB) and want to use Apriori and FP-GRowth to extract rules (Associate Rule Mining). I read in somewhere on the internet, FP-Growth should be faster than Apriori ...
0
votes
0answers
15 views

Getting Error while predicting during deploying simple ML model with Flask

I have deployed a simple titanic (doing for testing purpose) model with Python Flask. I am using below server.py (Flask app start and predict API code) and request.py (JSON request) code. Getting ...
1
vote
0answers
26 views

How to detect Class label in Apriori algorithm in MATLAB?

Sorry if my my question is simple. I just would like to know how to detect class labels in data mining algorithms such as Apriori. As I know, in the normal use of the Apriori algorithm our patterns ...
1
vote
0answers
7 views

Could perceptron use the standard logistic function as its activation function?

the perceptron is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to ...
0
votes
0answers
20 views

What could be the source of a systematic error in prediction?

While training a LSTM model on time series data I am getting a constant over prediction as below. The features are properly scaled and there is no evident over-fitting as checked from learning curves....
2
votes
0answers
13 views

pros and cons of lexical vs machine learning methods for text mining

I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic. I have used a simple method of mining documents related to a ...
0
votes
0answers
9 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
0
votes
0answers
17 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
8 views

Analysis on the 5 segment of the data to differentiate from each other using R

I have 1000 customers operational data which is already divided into segments. Now I have to do analysis on different segments to find out how they differ from each other. Why they fall into their ...
2
votes
3answers
54 views

Visualize/analyze data before or after imputing missing values?

My understanding is that we impute missing values in order to preserve those training examples so our Machine Learning algorithms have as many training examples as possible. To me, it would make ...
1
vote
0answers
16 views

How to perform Learning to Rank for a small dataset

I am very interested in applying Learning to rank to my problem doamin. When I read through the literature of Learning to rank I ...
0
votes
0answers
12 views

Extract data from mainly unstructured sets and derive risk metrics out of those

I have the following question (this was a real life example): Q: Extract data from mainly unstructured sets and derive risk metrics out of those. From what you know or imagine about the data ...
0
votes
0answers
16 views

Class Size Imbalance for LDA or any other Content based analysis

I am running some content analysis studies on my dataset which has two different classes, and each class has a respective list of the document I am analyzing. I compare the LDA topic model inference ...
0
votes
1answer
68 views

Predict Customer Next Purchase with Sequence

Suppose I buy products: [1,2,3,4] Another customer X bought: [2,3] Most probably customer X next purchase will be: 4 Sequence is very important in my problem I tried association analysis using R, ...
1
vote
1answer
23 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
-1
votes
1answer
24 views

Extracting information from online job postings

Okay, so I'm trying to build a data set about data science job openings. I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
2
votes
2answers
48 views

Is there a real life meaning about KMeans error?

I am trying to understand the meaning of error in sklearn KMeans. In the context of house pricing prediction, the error linear regression could be considered as the money difference per square foot. ...
2
votes
0answers
132 views

Can we use log transformation for outliers?

I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions, 1) When I skewness and kurtosis for the normality test, two ...
0
votes
0answers
11 views

Is a transaction with 1 item important in apriori?

Im doing association analysis. I have alot of transactions with only 1 item Are they important or should I remove them? because transactions of 1 item doesnt really let you know , if item 1 bought ...
0
votes
0answers
11 views

Identifying patterns/motifs in categorical time series

There is a data set I currently possess of the following form: $D = [(s_0, t_0), (s_1, t_1), ..., (s_N, t_N)]$, where $s_i \in \mathcal{S}, 0 \leq t_i < T$. Here $\mathcal{S}$ is a finite alphabet ...
0
votes
2answers
36 views

Convert non-binary value into binary value?

I'm kind of a newbie in ML I have a dataset like: ...
0
votes
1answer
20 views

Is it possible to build a regression model for predicting movie gross using sections on their wikipedia pages?

I got this as an assignment from a company recruiter and I've successfully scraped a dataset of about 650 movies with their 'Plot', 'Music' and 'Marketing' sections and gross. I've tried tfidf and ...
0
votes
1answer
51 views

Implication of a dominant Principal Component in PCA analysis

I need help, are there any practical implications of a dominant principal component. For example, if of three PCs, PC1 explains almost 100% of the variance in this dataset, What does this mean in ...
0
votes
0answers
63 views

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file

CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar', 'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site- packages\tabula\tabula-1.0....
1
vote
1answer
65 views

Why does np.linalg.eig produce an opposite-signed eigenvector?

I am learning SVD by following this MIT course. In this video, the lecturer is finding the SVD for $$ \begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix}, $$ which involves finding the ...
1
vote
1answer
101 views

is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

I am learning SVD by following this MIT course The lecturer is trying to normalize a vector $${\begin{bmatrix} -3\\ 1\\ \end{bmatrix}}$$ to $${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\...
1
vote
0answers
38 views

Speed of SVM.. Poly kernel method runs endlessly and never completes execution

I'm using scikitlearn in Python to create some models while trying different kernels. I was surprised to see that rbf was fit in under a second, whereas linear took a minute and poly took hours. Can ...
0
votes
1answer
58 views

Why n-split is not possible for a dataframe with KFold?

On running below code on python 3.7, I am getting the following response: 'DataFrame' object has no attribute 'n_splits'. How to get rid of this? ...
0
votes
0answers
127 views

Reading pdf file with all pages in Python error: CalledProcessError:

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file. ...
2
votes
1answer
33 views

How valuable is a categorical feature that has a predominant category over all other ones?

Is a categorical feature that has almost equally distributed in it's category more important or the one which one of it's category is predominant over all other ones? In data prepossessing step for "...
0
votes
1answer
36 views

What are good practices in reporting RMSE or MAPE estimates for a machine learning model?

I have built a regression model to estimate prices given some attributes. Prices are expressed in million USD, with average of 4.4 million USD. I need to provide an estimate of the RMSE and MAPE of ...