Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
372
questions
0
votes
0
answers
9
views
Metric for correlation of two paired lists of numbers
I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case:
...
0
votes
0
answers
6
views
Stacking ensembles in meta learning with only one base algorithm
I'm learning stacking ensembles in meta learning and , there is an example where thy used only lightgbm as base model and linear regression as meta model, they first Split thé dataset into 50 samples ...
0
votes
0
answers
20
views
Library for Phi correlation coefficient in python?
I want to calculate correlation b/w categorical features in my data. I reviewed the literature and found phi coefficient can be used for this purpose. I found one library called phik in python enter ...
- 27
0
votes
0
answers
21
views
Does lightGBM handle multicollinearity? [duplicate]
I have a dataset after feature selection of around 6500 features and 10,000 data rows. I am using LightGBM model. I want to know if I should check the feature set for multicollinearity. If two or more ...
- 27
1
vote
1
answer
24
views
Temperature lag forecasting
I am working on a data science project on an industrial machine. This machine has two heating infrastructures. (fuel and electricity). It uses these two heatings at the same time, and I am trying to ...
- 23
0
votes
0
answers
6
views
ML methods for vector correlation
I am dealing with a timeseries consisting of input flow sampled every 5 minutes over 441 days. My aim is to find any possible correlation from data coming from:
The same day of the week
The same ...
- 1
0
votes
1
answer
25
views
How to set the same number of datapoints in the different ranges in correlation chart
I am beginner in working with machine learning. I would like to ask a question that How could I set the same number of datapoints in the different ranges in correlation chart? Or any techniques for ...
0
votes
2
answers
10
views
Correlations with NA or with Zeros?
When calculating correlations in R e.g. via cor is it better to treat missing data as NAs or as ...
- 480
2
votes
2
answers
27
views
Treating highly correlated features to the label feature
We work on a dataset with >1k features, where some elements are temporal/non-linear aggregations of other features.
e.g., one feature might be the salary ...
- 23
0
votes
1
answer
85
views
Python: calculate the weighted average correlation coefficient
I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
- 101
0
votes
0
answers
7
views
distinct Correlation and Causation of data
i have a lab to find a cause-effect relationship between 2 columns of a data.
First i want to ask: causation is correlation, i mean that causation is subset of correlation. I ask this cause i saw ...
0
votes
1
answer
76
views
Dropping highly correlated features
I am making my classification project and I have this situation after using seaborn heatmap.
Column 0 is my target, where I have data with 3 classes. To my knowledge I should remove column highly ...
- 41
0
votes
0
answers
10
views
Filling NaN values
According to my knowledge, before filling nan values we have to check whether data is missing because of MCAR, MAR or MNAR and it depends on how features are correlated with each other and then make a ...
0
votes
0
answers
11
views
does R2 diverge because of a lack of input dimensions?
I try to improve my R2 score between theoretical and real output values. On the picture you can see two cases: the blue one is an artificial case I’m completely mastering with 7 dimensions as input ...
- 73
0
votes
0
answers
13
views
Looking for data analysis techniques and approach
I'm new into ML and I need to do a data analysis on a dataset which I created myself but I don't know what techniques should I use exactly. Namely, I have a dataset with the following attributes: <...
- 101
0
votes
0
answers
15
views
How to explain this coorelation between features
Can somebody please help how to explain this correlation between features, as it does not have linear coorelation, but still seems to have somewhat coorelation. Here is the screenshoot:
- 11
0
votes
0
answers
34
views
Log odds understanding
Here is my understanding of one reason why we prefer log odds over odds & probability. Please let me know if I got it right.
Reasons why we choose log-odds-
The range of probability values: $[0,1]$...
- 239
0
votes
0
answers
9
views
Which statistical tests can I conduct to analyse the trend of series data?
I have a dataset that measures the a student time spent working on a mathematics question. My dataframe looks a little something like this:
Participant ID
Question 1
Question 2
Question 3
1107
54.2
...
- 101
0
votes
1
answer
16
views
Correlation Wikipedia translated pages vs number of in links is weird (scatterplot)?
I'm trying to find a correlation measure for the number of Wikipedia pages an entity (an article) has been translated to vs number of links that point to that page (both measures that can point to the ...
0
votes
0
answers
16
views
Correlation analysis yields conflicting results. Positive Pearson and Negative Spearman
I have four features x1,x2,x3,x4 all of their correlation with y are similar in Pearson and in Spearman separately. However, all these are +0.15 in Pearson and -0.6 in Spearman, more or less. Does ...
0
votes
0
answers
10
views
How to model multivariate time series when target looks random?
I am trying to model a multivariate time series to predict a target variable. The target vs time plot looks like this.
Here is the autocorrelation plot of the target versus lag.
As you can see the ...
0
votes
0
answers
17
views
Na values in rankings
Having a table of rankings containing many Na values, how should I deal with Na values while calculating the correlation between those rankings?
- 51
0
votes
1
answer
20
views
What correlation measure for Wikipedia translated pages vs number of in links?
I'm trying to find a correlation measure for the number of Wikipedia pages an entity (an article) has been translated to vs number of links that point to that page (both measures that can point to the ...
2
votes
1
answer
29
views
Rank correlation with spearman and kendall
While interpreting the correlation between ranks, should I use the rho value (for spearman method), tau value (for kendall's tau method), w value ( for kendall's w method) or should I take in ...
- 51
1
vote
0
answers
62
views
Are predictive features with 0 SHAP values included in the model?
I have trained and XGBoost by enforcing no-feaure interaction and calculated Global Shap values:
It looks like only 6 features have some SHAP values, whilst the remaining ones have a SHAP value of 0.
...
0
votes
0
answers
103
views
Why are SHAP values not an indication of cause?
I have trained an XGBoost Classifier and I am now trying to explain how and, most importantly, why the model has made the predictions it's made.
In the documentation entry Be careful when interpreting ...
1
vote
1
answer
24
views
Is it possible for a feature not correlated with a dependent variable to become important in a machine learning model?
Is it possible for a feature not correlated (or faintly correlated) with a dependent variable to become important in a machine learning model?
- 209
0
votes
0
answers
15
views
Why we need a statistical hypothesis testing for correlations between variables when we can check using scatter plots?
Need Guidance on correlation test.
When we need to perform correlation between 2 variables, we generally start with scatter plot. Sometime it is suggested to perform hypothesis testing as well.
I'm ...
- 109
0
votes
0
answers
50
views
Is pearson correlation matrix a good indicator for label encoded categorical and numeric independent data?
I have a dataset having 22 independent variables out of which 15 are categorical data that has already been label encoded i.e the dtype is int64 and the contents are in a range of 0 to n (n is the ...
- 103
0
votes
0
answers
20
views
Need help with simple data analysis on Excel source
Apologies beforehand, this is outside my comfort zone.
I have a table in Excel with a list of ca. 100 items. Each of the items has a number of tags associated with it, indicated by a TRUE value in one ...
1
vote
1
answer
38
views
Is it possible to explain why Lasso models eliminated certain coefficient?
Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
- 209
1
vote
1
answer
42
views
Understanding which variables impact your variable of interest the most (correlation, linear regression) and correctly interpreting results
How do you ascertain which variables lead to the greatest increase in another variable of interest?
Let's say you have a correlation matrix. You look at the row of the variable you are particularly ...
0
votes
1
answer
40
views
how to confirm a correlation between features
i have created a matrix to show the correlation between different features in this data. i do not believe my findings. surely the blp (price to buy the car), should correlate with the rental price ...
0
votes
0
answers
31
views
Does the sign of correlation matter in feature selection?
If I understand correctly, the correlation between features and the target can be used to quantify whether those features are relevant to keep, hence the ritual of plotting the correlation matrix as a ...
- 175
0
votes
1
answer
32
views
Correlation with target variable for regression problem
Given the following dataframe
age job salary
0 1 Doctor 100
1 2 Engineer 200
2 3 Lawyer 300
...
with ...
- 705
0
votes
0
answers
21
views
What does a lag of 2N mean in autocorrelation?
From what I understand, when performing auto-correlation we compute the correlation of a signal with its own shifted version at different shift times (1,2 ..) which is referred as the lag. Now if my ...
- 1
1
vote
1
answer
49
views
Why is an ML algorithm performing better with correlated features, than the one with uncorrelated ones?
I have a dataset with all numerical values. Since the features were not many, I created more by multiplying pairs of each other. This created some highly correlated features, as expected. Now, I ...
1
vote
0
answers
18
views
Approach for finding patterns in daily event data
I am software engineer but have zero data science background, so apologies for this basic question.
I would like to find correlations between different behaviors of my daily life and different ...
- 111
0
votes
0
answers
25
views
what are the effect on machine learning regression model if the dataset has two exact same columns
What will be the effect on the Machine learning model if the dataset has two exact same columns(exact 1 correlation).
One thing that comes to my mind is that if two columns are exactly the same then ...
0
votes
0
answers
23
views
What correlation is considered to be big for linear regression predictors?
It is well known that if two linear regression predictors highly correlate, it is bad for our model, but which correlation is considered to be big? Is it 0.5,0.6,0.8,0.9..? I have tried to find out ...
- 11
0
votes
1
answer
38
views
Analysis with categorical variable
My dataset consists of a numeric variable (called "N4") and several categorical variables that affect the numeric variable. For example there is a categorical variable called "die" ...
- 459
1
vote
0
answers
39
views
statistical tests for null hypothesis - what if model is non linear?
I am reading the "An Introduction to Statistical Learning" (Gareth James & alii, Springer) as a primer to machine learning.
I am reading the part in linear regressors, and learnt there ...
- 155
0
votes
0
answers
17
views
Best method for calculating all correlations in a dataset?
First of all, to be clear, I don't want all correlations in the dataset, I just want to find all high correlations between the response variable and the features.
I posted this question to reddit on '...
- 151
1
vote
0
answers
24
views
Feature creation: Problem with correlated features?
I recently started to read about feature creation. I've seen some general guidelines although I am not really sure if they are completely true, for example:
1 - Linear classifiers for binary ...
- 111
1
vote
1
answer
35
views
Why is pearson correlation popular if it detects only linear correlation?
Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
- 15
1
vote
1
answer
44
views
Find feature categories associated with a specific target category
I have a Dataset with three columns.
Products (up to 200).
The quality checks that have not been conducted at the final quality check. (Up to 70 different Quality Control Measures)
The result of the ...
- 21
0
votes
0
answers
59
views
Predict multiple labels when labels are of mixed type: floating point and categorical
I was provided with a data file which looks like this:
...
0
votes
0
answers
23
views
Is there a "tree-based-correlation" for tree-based algorithms?
Although correlated features are not a big issue when training tree-based models, they spoil model explainability. When several features correlate, sometimes they may be picked at random. Then their ...
- 164
1
vote
1
answer
20
views
How to test whether data is clustered wrt. subcategories?
I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
user126321
0
votes
0
answers
12
views
Event correlation in processes
I have a dataset with roughly 100 events per product (150 products). So each product has these 100 events. The last event is a quality check which is either red or green and which is either due or on ...
- 21