The 2024 Developer Survey results are live! See the results

# Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

402 questions
Filter by
Sorted by
Tagged with
21 views

### cor() function in R returns NA when given two integers or continuous values

Using R version 4.4.0, I was using the cor() function and originally inputting two continuous values to obtain a default calculated correlation. NA is returned instead of a numeric. I'm able to ...
13 views

### Hash function to decorrelate datasets

Consider three sets of data A, B and C. mutual information I(A:B)=I(A:C). Now I apply hashing function to A to obtain a new data A' with entropy H(A|B). does that imply I(A':C)=0.
9 views

### how to properly assess correlation in my data

I have a dataset that contains football events. It provides the following info: event type (on target, off target, goal) event sub-type (header, distance shot...) player's name player's height (in cm)...
16 views

### Understanding most important features from an additional column

I'm fairly new to data science in general and I'm doing some analysis. Let us say I have N rows and D features, and I have a ...
29 views

### pattern within autocorrelation confidence intervals

I have a time-series producing the following auto & partial correlation plots. What insights can we make when there are oscillatory patterns within the grey region of insignificance? Does it make ...
• 351
21 views

### How to check if an event affects time series

We have time series data. Depended variable – interest rates, about 15 years, monthly data. Independent variable – event, rating announcement (rating may change or may not), happens 2-3 times per year,...
38 views

### Correlation between predictions vs correlation between targets

In a multi-target model framework - where a separate model is estimated for each target - how can one take into account for correlations between targets during the training process ? For example say I ...
• 121
97 views

### Pearson correlation with overlapping data

I have a financial time series and I want to calculate correlation between past and future returns. First I select look back and holding periods, say l and h respectively. Then I calculate past ...
14 views

### How to visualize correlation values coming from two data frame for comparison

I am working on a project wherein we are comparing two methods used for modeling gene expression. One method is using elastic net and other is using lasso regression. In one method: we see that ...
20 views

### Ways to check whether your predictors are even correlated with their label?

I scraped a dataset of pre-match data in a video game and am trying to classify them by outcome, i.e. wins v. losses. The models I've tried so far have poor accuracy of around 50%, so I'm thinking ...
38 views

### Measuring features effect and importance in Partial Least Square (PLS) regression

Context: it is possible to assess features importance and effect for a model using model-independent scoring techniques such as Partial Dependence (PD) profile, Acculumated Local Effect (ALE) profile, ...
• 101
143 views

### Calculating correlation between embedding features

I am looking for a technique to calculate correlation for embedding features (array of floats). I'm interested in the correlation between features (embedding-embedding) as well as between feature and ...
13 views

### Correlation in forecast error

I am working on a time series project and while calculating the prediction intervals using bootstrapping method I have to fulfill the assumptions of it. It has an assumption of uncorrelated forecast ...
54 views

### TSNE plots of random data subsets are vastly different but labels are still clearly separated - what conclusions can we draw about the dataset?

I scraped a dataset of match data in a video game and labeled them according to their outcome (0 for loss, 1 for win). I wanted to see if there was actually any inherent relationship between the ...
171 views

### Correlation Analysis dealing with missing value, when the missing values are actually expected

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here ...
10 views

### What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
• 21
17 views

### Good candidate time series models for 70-100 monthly data points, also incorporate past and future exogenous covariates

Per title, we're trying to identify a good time series modeling technique for: 70-100 variables of monthly sales or volume data (2015 or 2018 to present) Ability to forecast not only using trend and ...
• 101
67 views

### Correlation in R between categorical and boolean variables

I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
• 1
110 views

### Why does the order of columns in CCA change some results?

When running CCA from scikit learn, if the order is changed in the columns like (changing the order of the rows of both datasets together does not produce different results that I've seen -- not shown ...
• 101
46 views

### What is reliability? How it is related to correlation coefficient?

We can compute real or population correlation(rho) by square-root of 1 minus R-squared.Is this a correct interpretation? Does population correlation mean a real correlation measured as square-root of ...
48 views

### Need feedback on idea for new regularization term

I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of ...
90 views

### Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
61 views

### Pearson coefficients different for same data?

Is it possible to have different Pearson coefficients over the same input data?
• 17
164 views

### Edit friendly DDPM noise space

I was reading this paper, "An Edit Friendly DDPM Noise Space: Inversion and Manipulations". In page no. 4, they have mentioned that in DDPM, noise maps of consecutive steps are highly ...
• 140
43 views

### Automatic statistical analysis / statistical "diff" tool?

I'm new to data science and wondered if there are ways to have a tool figure out relations between variables that may be relevant to a problem. Imagine I have a log file that I have pre-processed to ...
231 views

2k views

### How does Pandas' Correlation Method Handle Non-Numeric Columns?

I'm using Pandas' .corr() method to figure out which columns I can eliminate from a large dataset. Some of those columns have non-numeric types. How does Pandas ...
• 671
1k views

### How do you Speed up the Calculation of a Correlation Matrix on a Large Dataset in Pandas?

I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' ...
• 671
717 views

### What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?

To calculate the coefficient of partial determination R2 for a given variable: We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
• 185
106 views

### What is the fastest way to detect lag and calculate cross correlation of two binary time series?

Example, arr1 = array([0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0]) arr2 = array([1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0]) arr2 is almost perfectly correlated with ...
• 21
27 views

### Which redundant feature should be use

I have two redundant features. A & B with 0.85 correlation. I know only one of them should be used to trained my model, but which feature should i use? A or B? Is there any method that can i use ...
1k views

### Correlation vs Mutual Information vs let-the-model-decide

I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which ...
22 views

### Different Correlation Coefficents with different Time Ranges

I built a Time-Series that displays the price of the Electricty Price in South Italy and two of their most important commodities (commodities, gas) used to produce the eletrical energy. So I ordered ...