Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Metric for correlation of two paired lists of numbers

I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case: ...
user avatar
0 votes
0 answers
6 views

Stacking ensembles in meta learning with only one base algorithm

I'm learning stacking ensembles in meta learning and , there is an example where thy used only lightgbm as base model and linear regression as meta model, they first Split thé dataset into 50 samples ...
user avatar
0 votes
0 answers
20 views

Library for Phi correlation coefficient in python?

I want to calculate correlation b/w categorical features in my data. I reviewed the literature and found phi coefficient can be used for this purpose. I found one library called phik in python enter ...
user avatar
  • 27
0 votes
0 answers
21 views

Does lightGBM handle multicollinearity? [duplicate]

I have a dataset after feature selection of around 6500 features and 10,000 data rows. I am using LightGBM model. I want to know if I should check the feature set for multicollinearity. If two or more ...
user avatar
  • 27
1 vote
1 answer
24 views

Temperature lag forecasting

I am working on a data science project on an industrial machine. This machine has two heating infrastructures. (fuel and electricity). It uses these two heatings at the same time, and I am trying to ...
user avatar
  • 23
0 votes
0 answers
6 views

ML methods for vector correlation

I am dealing with a timeseries consisting of input flow sampled every 5 minutes over 441 days. My aim is to find any possible correlation from data coming from: The same day of the week The same ...
user avatar
0 votes
1 answer
25 views

How to set the same number of datapoints in the different ranges in correlation chart

I am beginner in working with machine learning. I would like to ask a question that How could I set the same number of datapoints in the different ranges in correlation chart? Or any techniques for ...
user avatar
0 votes
2 answers
10 views

Correlations with NA or with Zeros?

When calculating correlations in R e.g. via cor is it better to treat missing data as NAs or as ...
user avatar
  • 480
2 votes
2 answers
27 views

Treating highly correlated features to the label feature

We work on a dataset with >1k features, where some elements are temporal/non-linear aggregations of other features. e.g., one feature might be the salary ...
user avatar
  • 23
0 votes
1 answer
85 views

Python: calculate the weighted average correlation coefficient

I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
user avatar
  • 101
0 votes
0 answers
7 views

distinct Correlation and Causation of data

i have a lab to find a cause-effect relationship between 2 columns of a data. First i want to ask: causation is correlation, i mean that causation is subset of correlation. I ask this cause i saw ...
user avatar
0 votes
1 answer
76 views

Dropping highly correlated features

I am making my classification project and I have this situation after using seaborn heatmap. Column 0 is my target, where I have data with 3 classes. To my knowledge I should remove column highly ...
user avatar
  • 41
0 votes
0 answers
10 views

Filling NaN values

According to my knowledge, before filling nan values we have to check whether data is missing because of MCAR, MAR or MNAR and it depends on how features are correlated with each other and then make a ...
user avatar
0 votes
0 answers
11 views

does R2 diverge because of a lack of input dimensions?

I try to improve my R2 score between theoretical and real output values. On the picture you can see two cases: the blue one is an artificial case I’m completely mastering with 7 dimensions as input ...
user avatar
0 votes
0 answers
13 views

Looking for data analysis techniques and approach

I'm new into ML and I need to do a data analysis on a dataset which I created myself but I don't know what techniques should I use exactly. Namely, I have a dataset with the following attributes: <...
user avatar
  • 101
0 votes
0 answers
15 views

How to explain this coorelation between features

Can somebody please help how to explain this correlation between features, as it does not have linear coorelation, but still seems to have somewhat coorelation. Here is the screenshoot:
user avatar
0 votes
0 answers
34 views

Log odds understanding

Here is my understanding of one reason why we prefer log odds over odds & probability. Please let me know if I got it right. Reasons why we choose log-odds- The range of probability values: $[0,1]$...
user avatar
  • 239
0 votes
0 answers
9 views

Which statistical tests can I conduct to analyse the trend of series data?

I have a dataset that measures the a student time spent working on a mathematics question. My dataframe looks a little something like this: Participant ID Question 1 Question 2 Question 3 1107 54.2 ...
user avatar
  • 101
0 votes
1 answer
16 views

Correlation Wikipedia translated pages vs number of in links is weird (scatterplot)?

I'm trying to find a correlation measure for the number of Wikipedia pages an entity (an article) has been translated to vs number of links that point to that page (both measures that can point to the ...
user avatar
0 votes
0 answers
16 views

Correlation analysis yields conflicting results. Positive Pearson and Negative Spearman

I have four features x1,x2,x3,x4 all of their correlation with y are similar in Pearson and in Spearman separately. However, all these are +0.15 in Pearson and -0.6 in Spearman, more or less. Does ...
user avatar
0 votes
0 answers
10 views

How to model multivariate time series when target looks random?

I am trying to model a multivariate time series to predict a target variable. The target vs time plot looks like this. Here is the autocorrelation plot of the target versus lag. As you can see the ...
user avatar
0 votes
0 answers
17 views

Na values in rankings

Having a table of rankings containing many Na values, how should I deal with Na values while calculating the correlation between those rankings?
user avatar
  • 51
0 votes
1 answer
20 views

What correlation measure for Wikipedia translated pages vs number of in links?

I'm trying to find a correlation measure for the number of Wikipedia pages an entity (an article) has been translated to vs number of links that point to that page (both measures that can point to the ...
user avatar
2 votes
1 answer
29 views

Rank correlation with spearman and kendall

While interpreting the correlation between ranks, should I use the rho value (for spearman method), tau value (for kendall's tau method), w value ( for kendall's w method) or should I take in ...
user avatar
  • 51
1 vote
0 answers
62 views

Are predictive features with 0 SHAP values included in the model?

I have trained and XGBoost by enforcing no-feaure interaction and calculated Global Shap values: It looks like only 6 features have some SHAP values, whilst the remaining ones have a SHAP value of 0. ...
user avatar
0 votes
0 answers
103 views

Why are SHAP values not an indication of cause?

I have trained an XGBoost Classifier and I am now trying to explain how and, most importantly, why the model has made the predictions it's made. In the documentation entry Be careful when interpreting ...
user avatar
1 vote
1 answer
24 views

Is it possible for a feature not correlated with a dependent variable to become important in a machine learning model?

Is it possible for a feature not correlated (or faintly correlated) with a dependent variable to become important in a machine learning model?
user avatar
  • 209
0 votes
0 answers
15 views

Why we need a statistical hypothesis testing for correlations between variables when we can check using scatter plots?

Need Guidance on correlation test. When we need to perform correlation between 2 variables, we generally start with scatter plot. Sometime it is suggested to perform hypothesis testing as well. I'm ...
user avatar
0 votes
0 answers
50 views

Is pearson correlation matrix a good indicator for label encoded categorical and numeric independent data?

I have a dataset having 22 independent variables out of which 15 are categorical data that has already been label encoded i.e the dtype is int64 and the contents are in a range of 0 to n (n is the ...
user avatar
  • 103
0 votes
0 answers
20 views

Need help with simple data analysis on Excel source

Apologies beforehand, this is outside my comfort zone. I have a table in Excel with a list of ca. 100 items. Each of the items has a number of tags associated with it, indicated by a TRUE value in one ...
user avatar
1 vote
1 answer
38 views

Is it possible to explain why Lasso models eliminated certain coefficient?

Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
user avatar
  • 209
1 vote
1 answer
42 views

Understanding which variables impact your variable of interest the most (correlation, linear regression) and correctly interpreting results

How do you ascertain which variables lead to the greatest increase in another variable of interest? Let's say you have a correlation matrix. You look at the row of the variable you are particularly ...
user avatar
0 votes
1 answer
40 views

how to confirm a correlation between features

i have created a matrix to show the correlation between different features in this data. i do not believe my findings. surely the blp (price to buy the car), should correlate with the rental price ...
user avatar
0 votes
0 answers
31 views

Does the sign of correlation matter in feature selection?

If I understand correctly, the correlation between features and the target can be used to quantify whether those features are relevant to keep, hence the ritual of plotting the correlation matrix as a ...
user avatar
  • 175
0 votes
1 answer
32 views

Correlation with target variable for regression problem

Given the following dataframe age job salary 0 1 Doctor 100 1 2 Engineer 200 2 3 Lawyer 300 ... with ...
user avatar
0 votes
0 answers
21 views

What does a lag of 2N mean in autocorrelation?

From what I understand, when performing auto-correlation we compute the correlation of a signal with its own shifted version at different shift times (1,2 ..) which is referred as the lag. Now if my ...
user avatar
  • 1
1 vote
1 answer
49 views

Why is an ML algorithm performing better with correlated features, than the one with uncorrelated ones?

I have a dataset with all numerical values. Since the features were not many, I created more by multiplying pairs of each other. This created some highly correlated features, as expected. Now, I ...
user avatar
1 vote
0 answers
18 views

Approach for finding patterns in daily event data

I am software engineer but have zero data science background, so apologies for this basic question. I would like to find correlations between different behaviors of my daily life and different ...
user avatar
  • 111
0 votes
0 answers
25 views

what are the effect on machine learning regression model if the dataset has two exact same columns

What will be the effect on the Machine learning model if the dataset has two exact same columns(exact 1 correlation). One thing that comes to my mind is that if two columns are exactly the same then ...
user avatar
0 votes
0 answers
23 views

What correlation is considered to be big for linear regression predictors?

It is well known that if two linear regression predictors highly correlate, it is bad for our model, but which correlation is considered to be big? Is it 0.5,0.6,0.8,0.9..? I have tried to find out ...
user avatar
  • 11
0 votes
1 answer
38 views

Analysis with categorical variable

My dataset consists of a numeric variable (called "N4") and several categorical variables that affect the numeric variable. For example there is a categorical variable called "die" ...
user avatar
  • 459
1 vote
0 answers
39 views

statistical tests for null hypothesis - what if model is non linear?

I am reading the "An Introduction to Statistical Learning" (Gareth James & alii, Springer) as a primer to machine learning. I am reading the part in linear regressors, and learnt there ...
user avatar
0 votes
0 answers
17 views

Best method for calculating all correlations in a dataset?

First of all, to be clear, I don't want all correlations in the dataset, I just want to find all high correlations between the response variable and the features. I posted this question to reddit on '...
user avatar
  • 151
1 vote
0 answers
24 views

Feature creation: Problem with correlated features?

I recently started to read about feature creation. I've seen some general guidelines although I am not really sure if they are completely true, for example: 1 - Linear classifiers for binary ...
user avatar
1 vote
1 answer
35 views

Why is pearson correlation popular if it detects only linear correlation?

Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
user avatar
1 vote
1 answer
44 views

Find feature categories associated with a specific target category

I have a Dataset with three columns. Products (up to 200). The quality checks that have not been conducted at the final quality check. (Up to 70 different Quality Control Measures) The result of the ...
user avatar
0 votes
0 answers
59 views

Predict multiple labels when labels are of mixed type: floating point and categorical

I was provided with a data file which looks like this: ...
user avatar
0 votes
0 answers
23 views

Is there a "tree-based-correlation" for tree-based algorithms?

Although correlated features are not a big issue when training tree-based models, they spoil model explainability. When several features correlate, sometimes they may be picked at random. Then their ...
user avatar
  • 164
1 vote
1 answer
20 views

How to test whether data is clustered wrt. subcategories?

I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
user avatar
0 votes
0 answers
12 views

Event correlation in processes

I have a dataset with roughly 100 events per product (150 products). So each product has these 100 events. The last event is a quality check which is either red or green and which is either due or on ...
user avatar

1
2 3 4 5
8