Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
18 views

Calculation of data sparsity in data warehouse

suppose the data warehouse of an organisation contains 4000 customers 3000 products and only 500 sold per day to different customers? calculate the sparsity of data warehouse if we kept data for 10 ...
user avatar
  • 1
0 votes
0 answers
10 views

How to Calculate degrees in un-directed network and plot it? | Using python COLAB [duplicate]

I have loaded txt file (com-dblp.ungraph.txt) which includes DBLP collaboration network to a COLAB notebook. I found 317080 nodes & 1049866 edges in the network by reading the txt manually. Trying ...
user avatar
0 votes
0 answers
20 views

How to Calculate values inside DBLP collaboration un-directed network? | Using python

(EDITED- thanks) I have loaded txt file (com-dblp.ungraph.txt) which includes network to a COLAB notebook. I want to calculate describe vals of the net in the COLAB- how can I do it? Number of nodes ...
user avatar
0 votes
1 answer
13 views

Data Mining of unresearched data for a master's degree final project

So, I have to start thinking about the topic of my final project in a data science master's degree (business oriented, although I can choose any unrelated field) and one of the requirements is to ...
user avatar
0 votes
0 answers
9 views

Approach to check multicollinearity for data with categorical and continuous features?

I would like to know what should be the best approach to check for multicollinearity if my data has categorical and continuous variables like:- Age, Income, Department(more than 2 category), Gender(2 ...
user avatar
1 vote
1 answer
25 views

Interpreting cluster variables - raw vs scaled

I already referred these posts here and here. I also posted here but since there is no response, am posting here. Currently, I am working on customer segmentation using their purchase data. So, my ...
user avatar
  • 2,215
0 votes
1 answer
20 views

multi dimensional time series and matrix profile method

I have a time series of the following format: ...
user avatar
0 votes
0 answers
15 views

matrix profile distance measure characterization

If there are various types of distances measures for time series, such as Euclidean, DTW, and shape-based ones, how can we characterize the matrix profile distance measure? Profiling one?
user avatar
0 votes
0 answers
17 views

Finding data with transformation applied

Is there a way to find relatedness between data and the data obtained after some transformation applied to it? i.e. given a data I need to find the most related data(most of the values in that data ...
user avatar
  • 1
0 votes
0 answers
20 views

How can deep learning be applied to association rule mining?

Association rule mining is considered to be an old technique of AI. Rules are mined on statistical support. How can deep learning be applied to this? What are approaches for structured data (in a ...
user avatar
0 votes
0 answers
7 views

how to align sliding window to extract features from multi modal timeseries data?

I have two datasets that are collected at different frequencies at the same time. One is recorded at 128Hz and another one is recorded at 512 Hz. I am trying to extract some features using the moving ...
user avatar
0 votes
0 answers
9 views

Scaling and handling highly correlated features in tabular data for regression

I am working on a regression problem trying to predict a target variable with seven predictor variables. I have a tabular dataset of 1400 rows. Before delving into the machine learning to build a ...
user avatar
  • 101
0 votes
0 answers
15 views

How to do binning in matrix data

I have some data like ...
user avatar
0 votes
0 answers
17 views

Approaches on grouping/clustering network device data

So I come from more of a computer science background, and recently have been trying to find a solution to a data-centered problem. I would like to try experimenting different data-science methods on ...
user avatar
  • 1
0 votes
1 answer
18 views

How do I calculate the accuracy rate of predicting “Fail”? Am I supposed to create a confusion matrix?

Question: ABC Open University has a Teaching and Learning Analytics Unit (TLAU) which aims to provide information for data-driven and evidence-based decision making in both teaching and learning in ...
user avatar
0 votes
1 answer
35 views

What to do if the model is not performing well on a validation dataset

I am trying to use different ML classifiers for binary classification (SVM, logistic regression,DNN). The dataset used for training contains 333 columns and about 2000 rows. The classes being slightly ...
user avatar
1 vote
1 answer
52 views

Data preprocessing methods

Data Cleaning Data Imbalance solving (Classification) Data Smoothing (decreasing noise) Creating-deleting features from original data Data Transformation (Box-cox,Log Transform) Making Dataset ...
user avatar
  • 45
1 vote
1 answer
11 views

What is the formal category of problem described by identifying consecutive occurrences of attributes in records?

Apologies for the garbled title, I'd really need to know the answer to the question before I could phrase it properly... Let's imagine I've got a data set of football(soccer if you prefer) match ...
user avatar
2 votes
0 answers
28 views

Association between categorical variables with no hierarchy in Python

I have a dataset with over 100 possible variable occurrences across 20 columns. At first glance this problem seemed to fit into hierarchical clustering. I started testing with Agglomerative Clustering,...
user avatar
  • 21
1 vote
0 answers
60 views

Proof related to Ward's Method

According to Ward's Method that says :
user avatar
0 votes
0 answers
11 views

Clustering of time series data

I have a time series data set. I want to use Dynamic time warping for distance measurement. For algorithm, I was thinking of using either K-means DTW Barycenter Averaging (DBA) or K-medoids. Data has ...
user avatar
  • 1
1 vote
1 answer
23 views

Labelling for churn measurement

I have 3 domains of supplier data (Jan 2017 to Jan 2022) and they are as follows a) Purchase data - Contains all the purchase (of product) data made by the suppliers with us. It contains columns such ...
user avatar
  • 2,215
1 vote
1 answer
27 views

Confusion about the value of within-cluster SSE

I have a dataset of shape (29088, 11). When I apply the Kmeans where K=2 I get the following plot: I am surprised that the value of Sum Squared Error (SSE) for C0 (...
user avatar
  • 238
0 votes
0 answers
17 views

How to cluster the numerous variable with r studio?

How do I cluster the 18 variable with R studio? Should I pair the variable and cluster them? e.g pair Bounce rate and exit rate and cluster, pair visitor type and revenue and cluster. In this data set,...
user avatar
0 votes
1 answer
14 views

Comparing two groups at large scale

Let's consider we have two datasets. Dataset "A" and Dataset "B". Dataset "A" has two columns. Supplier_id and "Status" (pass and fail are values for status ...
user avatar
  • 2,215
0 votes
1 answer
38 views

Clustering - Auto ML Solutions

I intend to use clustering for my problem grouping customers together. However, instead of me manually tuning hyperparameters and CV, I would like to know is there anyway to do the clustering using ...
user avatar
  • 2,215
2 votes
0 answers
42 views

why the K-means optimization problem is non-convex?

K-means algorithm uses a residual sum of squares (RSS) where $RSS_{K} = \sum_{d \in s}|{d-c(s)}|^2$, $RSS = \sum_{k= 1}^{K}{RSS_{K}}$ is the convergence criterion. RSS is the objective ...
user avatar
  • 133
0 votes
0 answers
28 views

How to interpret calibration curves for prediction models?

I am working on a binary classification using random forest with 977 records (77:23 is the class ratio). After building the model and getting an AUC of 81, i thought of building a calibration curve ...
user avatar
  • 2,215
0 votes
1 answer
118 views

Should i always transform data to normal distribution?

I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
user avatar
  • 45
0 votes
0 answers
22 views

how to represent time series data as cross-sectional features

I already posted this here in cross-validated but since there is no response, am posting it here. I have a data like below subj_1 = 10,20,15,30,60,70,90 (in resolution of years - 2011 to 2018) subj_2 =...
user avatar
  • 2,215
1 vote
1 answer
27 views

Looking for in depth knowledge in evalution metric

I am dealing with an unbalanced dataset. The total instances in my dataset is 1273 and the Yes class is 174 and No class is 1099. So the unbalance ratio is like 1:6. Now I know ...
user avatar
  • 137
1 vote
0 answers
18 views

Intuitive explanation of FOIL's gain in Rule-based classification

I encounter the formula for calculating FOIL's gain as below: $$FOIL's\space gain = p_0(log_2(\frac{p_1}{p_1+n_1}) - log_2(\frac{p_0}{p_0+n_0}))$$ unlike Information gain or Gini index used to measure ...
user avatar
  • 11
0 votes
1 answer
44 views

Python - comparing 2 data frames [closed]

I have a large dataset, for an idea please see below. Each row has a sender, receiver, and message. There can be multiple columns for messages. It has indexes. sdr rcr msg A B msg_a A B msg_b A B ...
user avatar
  • 13
0 votes
0 answers
15 views

Data source for financial data mining

I plan to do data modeling in the financial area for my master's dissertation. I am thinking of finding the connection between a certain company or country characteristics ( x values) and their ...
user avatar
1 vote
0 answers
18 views

What is the formula of gradient boosting trees model?

I have been reading about gradient boosting trees (GBT) in some machine learning books and papers, but the references seem to only describe the training algorithms of GBT, but they do not describe the ...
user avatar
0 votes
0 answers
58 views

How to interpret the score output by a binary classifier when using a threshold < 0.5?

My understanding is that a score output by a binary classifier e.g. logistic regression for an input instance, is interpreted as the probability of the instance belonging to class 1. The threshold 0.5 ...
user avatar
0 votes
0 answers
13 views

Existing kpi vs New kpi as features to ML model

Am working on modelling the outcome - supplier met the target or not. For which, am currently working on feature engineering My business already has a set of performance criteria to indicate the ...
user avatar
  • 2,215
0 votes
0 answers
22 views

im working on movielens 100k dataset i would like to know can i assign cluster users based on k means or k prototype

I would like to know how to assign cluster id for users or movies for movielens100k dataset here is the link of dataset https://grouplens.org/datasets/movielens/100k/ I have a doubt like based on age ...
user avatar
1 vote
0 answers
177 views

SMOTE before categorical encoding vs SMOTE after categorical encoding

I have a small dataset of 977 rows with a class proportion of 77:23. For the sake of metrics improvement, I have kept my minority class ('default') as class 1 (and 'not default' as class 0). My input ...
user avatar
  • 2,215
0 votes
0 answers
8 views

Extracting data from human-to-human chat

I have a problem to solve and was hoping you could advise/point me in the right direction. The problem is: people returning products from my niche store talk to employees via a built-in chat. They ...
user avatar
0 votes
0 answers
5 views

What is the use of a 'supplemental' data

I have been working on a dataset from a drug chain supply company. It has one key dataset that has "historical data including Sales" (Sales is the target variable, that I ought to predict). ...
user avatar
1 vote
2 answers
165 views

Explainable AI solutions and packages in Python

I recently built a logistic regression for binary classification While I understand that logistic regression is inherentlt interpretable, I am trying to use explainable AI solutions to understand the ...
user avatar
  • 2,215
0 votes
2 answers
25 views

Auto-ML for only fixed estimator

Am working on a binary classification with 1000 rows and 28 columns. I would wish to use an Auto-ML solution to try out different combinations of hyperparameters etc but the algo should only be ...
user avatar
  • 2,215
1 vote
1 answer
35 views

how to link the predicted output to the original observation?

Am working on a binary classification using logistic regression data I have 1000 rows and 28 features. Three to 4 variables are Id variables like product_id, subject_id etc During train_test split, I ...
user avatar
  • 2,215
1 vote
1 answer
45 views

When it is okay to stick with low performance models?

I posted here already but it is marked to close, so thought of posting it here (as this might be the right forum) Am working on a simple logistic regression with 1000 records and 28 features. My ...
user avatar
  • 2,215
0 votes
0 answers
20 views

Live peak / trough detection (data provided)

At the bottom of this question is the data of three time series in CSV-format. All are of same length and they all contain measurements of the same event "A". But each time series is using a ...
user avatar
  • 101
0 votes
1 answer
21 views

Relationship of Bias and size of dataset

I was reading the following book: http://www.feat.engineering/resampling.html where the author mentioned the below: Generally speaking, as the amount of data in the analysis set shrinks, the ...
user avatar
0 votes
2 answers
56 views

Association Mining / rules are the statements appears to be true

I have a problem. I don't know how I cloud explain which of the three statements appears to be true and if my calculation below is invalid?` The following table summarizes the results of a medical ...
user avatar
  • 29
1 vote
3 answers
76 views

Association Mining - is buying Independent?

I have a problem. I can't not solve this exerciese. What is the best way to solve this exerciese? What are the approaches for this kind of exerciese? The following table summarizes transactions in a ...
user avatar
  • 29
2 votes
1 answer
51 views

exclude variables with no variation during prediction?

I am working on a binary classification problem. I do have certain input categorical variables such as gender, ethnicity etc. ...
user avatar
  • 2,215

1
2 3 4 5
23