Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
2
votes
1answer
26 views

Find Net Reclassification Improvement/Index metric using Python

I am working on a binary classification problem with ~5k records and class proportion of 33:67. I have 60 features/variables in my dataset and finally I have come to about 10 variables based on ...
2
votes
1answer
31 views

How to evaluate performance of a new feature in a model?

I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records. When I tried multiple models based on 6,7 and 8 features, I see the ...
8
votes
4answers
315 views

How to impute Missing values not the usual way?

I have a dataset of 4712 records working on binary classification. Label 1 is 33% and Label 0 is 67%. I can't drop records because my sample is already small. Because there are few columns which has ...
2
votes
1answer
24 views

Unable to do resampling using Python SMOTE

I am trying to do a simple ML re-sampling approach after the train-test split. However when I do this, it throws the below error. Can you please help me understand what is this error about? ...
1
vote
0answers
13 views

Need some advice on approach to select only the informative emojis from the data set?

I have a giant data set from a local elections, which contains hashtags, emojis, and comments. I wanted to make a network analysis using only emojis. So far I have a network analysis graph made in R ...
2
votes
0answers
9 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has 3 stocks, so I will have the 3x1 W (...
2
votes
1answer
23 views

Maximize one data point

I am completely new to data science and looking to narrow down the search and reduce the learning curve required to solve problems like the one given below I have a data set with 7 columns , Column ...
1
vote
1answer
36 views

ML Project - Achieve 2 Objectives

I have a dataset with 5K records focused on binary classification. I am posting it here to seek your suggestions on project methodology Currently what is my objective is 1) Run statsmodel logistic ...
2
votes
2answers
32 views

Which metric to choose for tracking model performance?

I am working on a binary classification problem with class proportion of 33:67. Currently what I am doing is running multiple models like LR,...
1
vote
1answer
39 views

How to select best feature set and not ranking for tree based models?

I am currently using feature selection approaches like filter, wrapper, embedded etc. All these methods give different set of ...
2
votes
2answers
293 views

Auto ML vs Manual ML for a project

I recently was introduced to a AUTO ML library based on genetic programming called tpot. Thanks to @Noah Weber. I have few ...
3
votes
1answer
228 views

feature selection using genetic algorithm in Python?

I have a dataset of 4712 records and 60+ features working on a binary classification problem. I already tried out all the feature selection approaches like ...
1
vote
2answers
52 views

How to perform bootstrap validation?

I am working on a binary classification problem. I ran cross-validation and grid-search on train data. Later I validated the model on my test data as shown below ...
0
votes
0answers
54 views

Why a significant risk factor doesn't increase AUC-score/F1-metric?

I have a binary classification problem with 5K records and 60+ features/columns/variables. dataset is slighlt imbalanced with 33:67 class proportion What I did was 1st) Run a logistic regression (...
1
vote
1answer
28 views

How to interpret statsmodel output - logit?

I ran a logit model using statsmodel api available in Python. I have few questions on how to make sense of these 1) What's the difference between ...
1
vote
2answers
25 views

Should I use MongoDB instead of storing data in CSV in python?

I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
2
votes
1answer
118 views

How to interpret dummy variable in ML prediction?

I am working on a binary classification problem where I have a mix of continuous and categorical variables. Categorical variables were created by me using ...
3
votes
0answers
14 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
3
votes
2answers
80 views

How to adjust cofounders in Logistic regression?

I have a binary classification problem where I apply logistic regression. I have a set of features that are found significant. But I understand that Logistic regression doesn't consider feature ...
2
votes
1answer
26 views

How to justify a predictor in influencing the outcome?

I am working on a prediction (binary classification) problem Currently I get an AUC score of 85-86 and F1-score of ...
1
vote
1answer
43 views

How to interpret shapley force plot for feature importance?

I am trying to practice and learn shapley value approach to explain my predictions on a binary classification problem. However am having difficulty in understanding the below plot. 1) Does it ...
1
vote
2answers
33 views

In datasets, why don't we represent nominal values that are part of a scale with numbers?

I am trying to pre-process a small dataset. I don't understand why I am not supposed to do the thing I explained below: For example, say we have an attribute that describes the temperature of the ...
0
votes
1answer
16 views

looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...
1
vote
1answer
34 views

Prediction vs causation in a ML project

I am performing a classification task and was able to identify significant predictors (important features using Random Forest) that can help separate the classes or influence the outcome. But I read ...
2
votes
3answers
62 views

Use of Standardizer to handle outliers?

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers. Actually I don't wish to drop the data as ...
0
votes
0answers
19 views

I am trying to implement Isolation forest for anomaly detection but I am not able to understand and visualise the decision function

I am trying to implement Isolation forest for anomaly detection but I am not able to understand the scores. Also, I want to plot and visualise the decision function that my Isolation forest is ...
3
votes
1answer
28 views

Data mining on reviews of the games

I am trying to text mine the reviews of the games and want to find the interesting things from the reviews example of the data is ...
1
vote
1answer
33 views

Role of a known predictor in a disease prediction

I am working on finding out whether the patient will develop the disease or not in a hospital. Might be a basic info but I am just sharing it anyway. Usually through historic data, I was able to see ...
1
vote
2answers
64 views

What's a good F1-score in (not) extremely imbalanced dataset?

I have a dataset with around 4.7K focused on binary classification. Class proportion is 33:67. meaning Label 1 is 1558 (33%) and Label 0 is 3154 (67%) of my dataset. Is my dataset imbalanced? some ...
0
votes
0answers
14 views
1
vote
2answers
53 views

How to get significance level for ranked features?

I am aware of below approaches of feature selection a) Feature Importance methods which are available in tree based models like Random Forest and ...
2
votes
1answer
309 views

How to interpret coefficients from logistic regression?

I ran a logistic regression (statsmodel) on my data with 60 features using the below code ...
1
vote
1answer
31 views

How to predict the dealer whether pick up the goods next month?

Here, I want to predict dealers(about 600) whether pick up the goods(about 30) next month. As you see, there are about 18,000 possibilities and it's difficult to predict. By the way, now I have ...
1
vote
1answer
53 views

How to perform Permutation Feature importance?

I am trying to perform feature selection. Currently with Tree based classifiers, even randomly generated column is ranking above some of my real columns. So I was reading about ...
3
votes
2answers
76 views

How to yield better AUC score?

I have a dataset with 5K records and 60 features focused on binary classification. Class proportion is 33:67 Currently I am trying to increase the performance of my model which is stuck at F1-score ...
1
vote
1answer
38 views

How to distinguish informative and non-informative feature - Feature importance?

I have a dataset with 5K records focused on binary classification problem. I have more than 60 features in my dataset. When I used Xgboost, I got the below ...
1
vote
1answer
50 views

How to Maximize recall for Minority class?

I have a dataset with 4.7k records and 60 features. 1558 records of indication label 1 and 3554 records indicating label 0. Am ...
1
vote
1answer
24 views

Mismatch between optimum features and grid scores using RFECV?

I have a dataset with 5K columns focused on binary classification. I have more than 60 columns. I am trying to find the best features through RFECV approach. Though it produces 30 optimum features, ...
0
votes
2answers
24 views

Dealing with data with no classes

I am new for data science subject and I am trying to learn on my own. Please bear with me if this is a stupid question. I have a dataset that has no classes. The data set comprises people's activity ...
1
vote
1answer
45 views

How to define the number of features to select in RFECV?

I am trying to work on feature selection stage for my dataset. I am a newbie to ML. I have around 60 columns and am trying to select top 15 features. I came to know about RFECV for which I wrote a ...
2
votes
2answers
160 views

How to select features for a ML model

I have a dataset with 5K records for binary classification problem. My features are min_blood_pressure, max_blood_pressure, <...
-1
votes
1answer
22 views

What are pickle, cPickle, and pickling in keras model?

I want to save full keras model. But in their documentation, they recommended not to use pickle or cPickle to to save a keras model. I am not clear about the two terms. What are they and why use or ...
0
votes
1answer
31 views

Evaluating likelihood of egg breaking when falling in random container on concrete [closed]

I am working on a project where I would like to predict whether an egg will break if it is put in a container that is then dropped on concrete. I am looking at the different factors that play a role ...
0
votes
0answers
23 views

Help required for medical imaging research | Deep Learning Project

I am a cs student currently working on Brain tumour segmentation using cascading of two U-Net research project. I have researched over the internet about the cascading of CNN but I found nothing about ...
0
votes
0answers
10 views

How far I can apply ARIMA forecast algorithm to my model?

I created a ARIMA model with 1224 month wise data (102 years - From Jan 1900 to Dec 2002), totalling to 1224 data items. How far I can forecast. Typically i got some pattern in forecast only for 36 ...
1
vote
1answer
44 views

Sequence to carry out data analysis?

I have a dataset with 4700 records and it's a classification problem. Proportion of classes is 33 and 67% few questions 1) does this proportion qualify dataset as imbalanced ? 2) should I do ...
0
votes
2answers
24 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
1
vote
1answer
18 views

On which step should use SMOTE technique for over sampling?

I want to use SMOTE technique for over sampling but I don't know on which step on pre-processing I should use it. My preprocessing steps are: Missing values Removing Outliers Smoothing Data Should ...
0
votes
0answers
27 views

What is the effect of changing the seed value on the classification results?

I used weka program to make a classification, first I used a dataset using the explorer and the seed value was equal to one, then I used the experimenter for the same dataset and the seed value was ...
0
votes
1answer
37 views

what does random seed value mean in weka?

I use weka to make a dataset classification but there is an option in classifier evaluation ( random seed for XVAL/% split). What does this option mean? ,and what is the seed value ? ,and What is the ...