# Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

372 questions
Filter by
Sorted by
Tagged with
9 views

### Metric for correlation of two paired lists of numbers

I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case: ...
6 views

### Stacking ensembles in meta learning with only one base algorithm

I'm learning stacking ensembles in meta learning and , there is an example where thy used only lightgbm as base model and linear regression as meta model, they first Split thé dataset into 50 samples ...
20 views

### Library for Phi correlation coefficient in python?

I want to calculate correlation b/w categorical features in my data. I reviewed the literature and found phi coefficient can be used for this purpose. I found one library called phik in python enter ...
• 27
21 views

### Does lightGBM handle multicollinearity? [duplicate]

I have a dataset after feature selection of around 6500 features and 10,000 data rows. I am using LightGBM model. I want to know if I should check the feature set for multicollinearity. If two or more ...
• 27
1 vote
24 views

### Temperature lag forecasting

I am working on a data science project on an industrial machine. This machine has two heating infrastructures. (fuel and electricity). It uses these two heatings at the same time, and I am trying to ...
• 23
6 views

### ML methods for vector correlation

I am dealing with a timeseries consisting of input flow sampled every 5 minutes over 441 days. My aim is to find any possible correlation from data coming from: The same day of the week The same ...
25 views

### How to set the same number of datapoints in the different ranges in correlation chart

I am beginner in working with machine learning. I would like to ask a question that How could I set the same number of datapoints in the different ranges in correlation chart? Or any techniques for ...
10 views

### Correlations with NA or with Zeros?

When calculating correlations in R e.g. via cor is it better to treat missing data as NAs or as ...
• 480
27 views

### Treating highly correlated features to the label feature

We work on a dataset with >1k features, where some elements are temporal/non-linear aggregations of other features. e.g., one feature might be the salary ...
• 23
85 views

### Python: calculate the weighted average correlation coefficient

I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
• 101
7 views

### distinct Correlation and Causation of data

i have a lab to find a cause-effect relationship between 2 columns of a data. First i want to ask: causation is correlation, i mean that causation is subset of correlation. I ask this cause i saw ...
76 views

### Dropping highly correlated features

I am making my classification project and I have this situation after using seaborn heatmap. Column 0 is my target, where I have data with 3 classes. To my knowledge I should remove column highly ...
• 41
10 views

### Filling NaN values

According to my knowledge, before filling nan values we have to check whether data is missing because of MCAR, MAR or MNAR and it depends on how features are correlated with each other and then make a ...
11 views

### does R2 diverge because of a lack of input dimensions?

I try to improve my R2 score between theoretical and real output values. On the picture you can see two cases: the blue one is an artificial case I’m completely mastering with 7 dimensions as input ...
13 views

### Looking for data analysis techniques and approach

I'm new into ML and I need to do a data analysis on a dataset which I created myself but I don't know what techniques should I use exactly. Namely, I have a dataset with the following attributes: <...
• 101
15 views

### How to explain this coorelation between features

Can somebody please help how to explain this correlation between features, as it does not have linear coorelation, but still seems to have somewhat coorelation. Here is the screenshoot:
34 views

### Log odds understanding

Here is my understanding of one reason why we prefer log odds over odds & probability. Please let me know if I got it right. Reasons why we choose log-odds- The range of probability values: $[0,1]$...
• 239
9 views

### Which statistical tests can I conduct to analyse the trend of series data?

I have a dataset that measures the a student time spent working on a mathematics question. My dataframe looks a little something like this: Participant ID Question 1 Question 2 Question 3 1107 54.2 ...
• 101
16 views

### Correlation Wikipedia translated pages vs number of in links is weird (scatterplot)?

I'm trying to find a correlation measure for the number of Wikipedia pages an entity (an article) has been translated to vs number of links that point to that page (both measures that can point to the ...
16 views

### Correlation analysis yields conflicting results. Positive Pearson and Negative Spearman

I have four features x1,x2,x3,x4 all of their correlation with y are similar in Pearson and in Spearman separately. However, all these are +0.15 in Pearson and -0.6 in Spearman, more or less. Does ...
10 views

### How to model multivariate time series when target looks random?

I am trying to model a multivariate time series to predict a target variable. The target vs time plot looks like this. Here is the autocorrelation plot of the target versus lag. As you can see the ...
17 views

### Na values in rankings

Having a table of rankings containing many Na values, how should I deal with Na values while calculating the correlation between those rankings?
• 51
20 views

### What correlation measure for Wikipedia translated pages vs number of in links?

I'm trying to find a correlation measure for the number of Wikipedia pages an entity (an article) has been translated to vs number of links that point to that page (both measures that can point to the ...
29 views

### Rank correlation with spearman and kendall

While interpreting the correlation between ranks, should I use the rho value (for spearman method), tau value (for kendall's tau method), w value ( for kendall's w method) or should I take in ...
• 51
1 vote
62 views

### Are predictive features with 0 SHAP values included in the model?

I have trained and XGBoost by enforcing no-feaure interaction and calculated Global Shap values: It looks like only 6 features have some SHAP values, whilst the remaining ones have a SHAP value of 0. ...
103 views

### Why are SHAP values not an indication of cause?

I have trained an XGBoost Classifier and I am now trying to explain how and, most importantly, why the model has made the predictions it's made. In the documentation entry Be careful when interpreting ...
1 vote
24 views

### Is it possible for a feature not correlated with a dependent variable to become important in a machine learning model?

Is it possible for a feature not correlated (or faintly correlated) with a dependent variable to become important in a machine learning model?
• 209
15 views

### Why we need a statistical hypothesis testing for correlations between variables when we can check using scatter plots?

Need Guidance on correlation test. When we need to perform correlation between 2 variables, we generally start with scatter plot. Sometime it is suggested to perform hypothesis testing as well. I'm ...
50 views

### Is pearson correlation matrix a good indicator for label encoded categorical and numeric independent data?

I have a dataset having 22 independent variables out of which 15 are categorical data that has already been label encoded i.e the dtype is int64 and the contents are in a range of 0 to n (n is the ...
• 103
20 views

### Need help with simple data analysis on Excel source

Apologies beforehand, this is outside my comfort zone. I have a table in Excel with a list of ca. 100 items. Each of the items has a number of tags associated with it, indicated by a TRUE value in one ...
1 vote
38 views

### Is it possible to explain why Lasso models eliminated certain coefficient?

Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
• 209
1 vote
42 views

### Understanding which variables impact your variable of interest the most (correlation, linear regression) and correctly interpreting results

How do you ascertain which variables lead to the greatest increase in another variable of interest? Let's say you have a correlation matrix. You look at the row of the variable you are particularly ...
40 views

### how to confirm a correlation between features

i have created a matrix to show the correlation between different features in this data. i do not believe my findings. surely the blp (price to buy the car), should correlate with the rental price ...
31 views

### Does the sign of correlation matter in feature selection?

If I understand correctly, the correlation between features and the target can be used to quantify whether those features are relevant to keep, hence the ritual of plotting the correlation matrix as a ...
• 175
32 views

### Correlation with target variable for regression problem

Given the following dataframe age job salary 0 1 Doctor 100 1 2 Engineer 200 2 3 Lawyer 300 ... with ...
• 705
21 views

### What does a lag of 2N mean in autocorrelation?

From what I understand, when performing auto-correlation we compute the correlation of a signal with its own shifted version at different shift times (1,2 ..) which is referred as the lag. Now if my ...
1 vote
49 views

### Why is an ML algorithm performing better with correlated features, than the one with uncorrelated ones?

I have a dataset with all numerical values. Since the features were not many, I created more by multiplying pairs of each other. This created some highly correlated features, as expected. Now, I ...
1 vote
18 views

### Approach for finding patterns in daily event data

I am software engineer but have zero data science background, so apologies for this basic question. I would like to find correlations between different behaviors of my daily life and different ...
• 111
25 views

### what are the effect on machine learning regression model if the dataset has two exact same columns

What will be the effect on the Machine learning model if the dataset has two exact same columns(exact 1 correlation). One thing that comes to my mind is that if two columns are exactly the same then ...
23 views

### What correlation is considered to be big for linear regression predictors?

It is well known that if two linear regression predictors highly correlate, it is bad for our model, but which correlation is considered to be big? Is it 0.5,0.6,0.8,0.9..? I have tried to find out ...
• 11
38 views

### Analysis with categorical variable

My dataset consists of a numeric variable (called "N4") and several categorical variables that affect the numeric variable. For example there is a categorical variable called "die" ...
• 459
1 vote
39 views

### statistical tests for null hypothesis - what if model is non linear?

I am reading the "An Introduction to Statistical Learning" (Gareth James & alii, Springer) as a primer to machine learning. I am reading the part in linear regressors, and learnt there ...
• 155
17 views

### Best method for calculating all correlations in a dataset?

First of all, to be clear, I don't want all correlations in the dataset, I just want to find all high correlations between the response variable and the features. I posted this question to reddit on '...
• 151
1 vote
24 views

### Feature creation: Problem with correlated features?

I recently started to read about feature creation. I've seen some general guidelines although I am not really sure if they are completely true, for example: 1 - Linear classifiers for binary ...
• 111
1 vote
35 views

### Why is pearson correlation popular if it detects only linear correlation?

Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
1 vote
44 views

### Find feature categories associated with a specific target category

I have a Dataset with three columns. Products (up to 200). The quality checks that have not been conducted at the final quality check. (Up to 70 different Quality Control Measures) The result of the ...
• 21
59 views

### Predict multiple labels when labels are of mixed type: floating point and categorical

I was provided with a data file which looks like this: ...
23 views

### Is there a "tree-based-correlation" for tree-based algorithms?

Although correlated features are not a big issue when training tree-based models, they spoil model explainability. When several features correlate, sometimes they may be picked at random. Then their ...
• 164
1 vote