Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

pattern within autocorrelation confidence intervals

I have a time-series producing the following auto & partial correlation plots. What insights can we make when there are oscillatory patterns within the grey region of insignificance? Does it make ...
eliangius's user avatar
  • 351
0 votes
1 answer
16 views

How to check if an event affects time series

We have time series data. Depended variable – interest rates, about 15 years, monthly data. Independent variable – event, rating announcement (rating may change or may not), happens 2-3 times per year,...
NoobinStatistics's user avatar
2 votes
0 answers
37 views

Correlation between predictions vs correlation between targets

In a multi-target model framework - where a separate model is estimated for each target - how can one take into account for correlations between targets during the training process ? For example say I ...
Kreol's user avatar
  • 121
0 votes
1 answer
56 views

Pearson correlation with overlapping data

I have a financial time series and I want to calculate correlation between past and future returns. First I select look back and holding periods, say l and h respectively. Then I calculate past ...
970541804's user avatar
0 votes
0 answers
13 views

How to visualize correlation values coming from two data frame for comparison

I am working on a project wherein we are comparing two methods used for modeling gene expression. One method is using elastic net and other is using lasso regression. In one method: we see that ...
Rhea Bedi's user avatar
0 votes
0 answers
18 views

Ways to check whether your predictors are even correlated with their label?

I scraped a dataset of pre-match data in a video game and am trying to classify them by outcome, i.e. wins v. losses. The models I've tried so far have poor accuracy of around 50%, so I'm thinking ...
Lilian Shi's user avatar
0 votes
0 answers
24 views

Measuring features effect and importance in Partial Least Square (PLS) regression

Context: it is possible to assess features importance and effect for a model using model-independent scoring techniques such as Partial Dependence (PD) profile, Acculumated Local Effect (ALE) profile, ...
Paul's user avatar
  • 101
0 votes
0 answers
80 views

Calculating correlation between embedding features

I am looking for a technique to calculate correlation for embedding features (array of floats). I'm interested in the correlation between features (embedding-embedding) as well as between feature and ...
Drew Serles's user avatar
0 votes
0 answers
10 views

Correlation in forecast error

I am working on a time series project and while calculating the prediction intervals using bootstrapping method I have to fulfill the assumptions of it. It has an assumption of uncorrelated forecast ...
Aakash Goyal's user avatar
0 votes
1 answer
30 views

TSNE plots of random data subsets are vastly different but labels are still clearly separated - what conclusions can we draw about the dataset?

I scraped a dataset of match data in a video game and labeled them according to their outcome (0 for loss, 1 for win). I wanted to see if there was actually any inherent relationship between the ...
Lilian Shi's user avatar
0 votes
2 answers
64 views

Correlation Analysis dealing with missing value, when the missing values are actually expected

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here ...
Sijan Bhattarai's user avatar
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
Imen F's user avatar
  • 21
0 votes
0 answers
17 views

Good candidate time series models for 70-100 monthly data points, also incorporate past and future exogenous covariates

Per title, we're trying to identify a good time series modeling technique for: 70-100 variables of monthly sales or volume data (2015 or 2018 to present) Ability to forecast not only using trend and ...
Vaslo's user avatar
  • 101
0 votes
0 answers
59 views

Correlation in R between categorical and boolean variables

I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
Max's user avatar
  • 1
0 votes
1 answer
105 views

Why does the order of columns in CCA change some results?

When running CCA from scikit learn, if the order is changed in the columns like (changing the order of the rows of both datasets together does not produce different results that I've seen -- not shown ...
user14094230's user avatar
0 votes
0 answers
37 views

What is reliability? How it is related to correlation coefficient?

We can compute real or population correlation(rho) by square-root of 1 minus R-squared.Is this a correct interpretation? Does population correlation mean a real correlation measured as square-root of ...
Subhash C. Davar's user avatar
0 votes
0 answers
48 views

Need feedback on idea for new regularization term

I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of ...
Brett L's user avatar
0 votes
1 answer
81 views

Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
Brett L's user avatar
0 votes
0 answers
20 views

ACF and PACF Autocorrelation

I have been trying to Understand correlation and tried it on my project and need help interpreting for a my case and in general. I have the following results but finding it hard to interpret. click ...
Angie's user avatar
  • 1
0 votes
0 answers
79 views

Correlation Matrix for Very Large Data Set

I am tasked to create a correlation matrix on a large portion of an MRI imaging data set (all numeric entries). The dimensions of the data is around 28,000x1,800. I have already set a correlation ...
user123's user avatar
  • 101
0 votes
1 answer
55 views

Pearson coefficients different for same data?

Is it possible to have different Pearson coefficients over the same input data?
SSSOF's user avatar
  • 17
2 votes
2 answers
149 views

Edit friendly DDPM noise space

I was reading this paper, "An Edit Friendly DDPM Noise Space: Inversion and Manipulations". In page no. 4, they have mentioned that in DDPM, noise maps of consecutive steps are highly ...
shivani's user avatar
  • 140
0 votes
1 answer
43 views

Automatic statistical analysis / statistical "diff" tool?

I'm new to data science and wondered if there are ways to have a tool figure out relations between variables that may be relevant to a problem. Imagine I have a log file that I have pre-processed to ...
Evgeniy Berezovsky's user avatar
0 votes
1 answer
190 views

How to find the correlation coefficient of one piece of data in a set?

I have a set of two pieces of data with a 0.85 coefficient. How can I calculate the coefficient for each individual set of data? For example, one set is 0-200 (x) and the other one is 0-$500,000 (y)....
Dave Bell's user avatar
1 vote
1 answer
73 views

How to proceed when correlation plot doesn't show as much multicorrelation as is seen by statsmodels variance_inflation_factor in a regression task?

I am working with the kaggle Blueberry Yield prediction dataset. There are 17 columns including the target variable. Below is the correlation heat map: It can be seen that multiple features are ...
forest's user avatar
  • 111
0 votes
0 answers
28 views

How can I correlate different types of variables?

first of all, my knowledge on this subject is very limited, therefore it may be a silly question. I have 3 vectors, two of them are (distance vectors) in the range of 0 and 1, and the other is (...
Mustyby2's user avatar
0 votes
0 answers
61 views

What is degree assortativity coefficient of a complete undirected graph?

Because it computes the correlation coefficient of degrees and the correlation coefficient of constant arrays is not defined, networkx library returns ...
Neo's user avatar
  • 3
0 votes
2 answers
2k views

corr() is giving an error. please help out of this problem and tell me what is this error about

when I am trying to run sns.heatmap(df.corr(),annot=True) this code in my jupyter notebook. this error is occuring. I cannot understand this problem. please help me.
Subhajit Sarkar's user avatar
0 votes
1 answer
95 views

Categorical feature relationship with target

When I'm working on dataset and want to explore different relationships between features and target, I often use visualization, only for regression with continous feature and continous target I use ...
jxqbbb's user avatar
  • 51
2 votes
1 answer
1k views

If my target variable is binary, is it better to use Pearson's or Spearman's for my correlation vector?

I'm using a corr vector, combined with RFE, to perform feature selection. I keep reading conflicting things online as to whether I should use Pearson's or Spearman's...
CodingNewbie's user avatar
0 votes
1 answer
45 views

Maximizing minimum correlation

What meaning has the weighted sum of a group of variables so that each weight is assigned to maximize the minimum resulting correlation of all these variables to the sum obtained?
gabriel's user avatar
0 votes
2 answers
331 views

How do we distinguish between correlated and un-correlated features/variables ? Is it relevant for a regression analysis?

Correlated and un-correlated terms are frequently used in data-science and understood as if they represent correlation coefficient. Is it the right way?
Subhash C. Davar's user avatar
1 vote
1 answer
40 views

Best way to detect newly incoming anomalies in two timeseries?

I have two devices that both send data (let's say temperature). I need to be able to detect if one of the devices reports an unusual/anomalous reading. The case if both of the devices report a "...
Jamess11's user avatar
0 votes
1 answer
439 views

Should highly correlated features be removed, even if they have different type of information?

A quick example for this: we have many feature and two of them are policy count and premium_total (for all policies). We are predicting the expected claim amount with GBM or RF. Both policy_count and ...
morqueatsz's user avatar
0 votes
1 answer
39 views

Best way to compare classification output between different locations

I ran a neural network for 20+ different locations across the United States. At each location I have a list of their predictions in an array. This looks something like this... ...
Jack Cahill's user avatar
0 votes
1 answer
786 views

Shapley Values - How to interpret each value for each feature for a specific instance?

I am using Shap Values(the 'shap' module in python) to help me understand a bit better the relation between my features and my target. I am currently working on a binary classification problem. I know ...
Gabriel Monteiro's user avatar
0 votes
1 answer
183 views

How can I take the cross-correlation between two discrete signal with slightly different discrete time points?

I have two discrete signals x1 and x2 with corresponding time points t1 and ...
hokge's user avatar
  • 1
0 votes
1 answer
129 views

How to correctly compute the correlation index of a column value from table in Python 3?

I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
janjilecek's user avatar
0 votes
1 answer
62 views

Generate a set of values that has a given correlation with n given other sets of values

Given lists $L_1, \dots, L_n$ of, let's say, 2000 values each and arbitrary numbers $c_1, \dots, c_n$, is it possible to generate a random list of 2000 values that has correlation $c_i$ with $L_i$ for ...
akuea3728's user avatar
0 votes
1 answer
2k views

How does Pandas' Correlation Method Handle Non-Numeric Columns?

I'm using Pandas' .corr() method to figure out which columns I can eliminate from a large dataset. Some of those columns have non-numeric types. How does Pandas ...
Connor's user avatar
  • 651
2 votes
1 answer
1k views

How do you Speed up the Calculation of a Correlation Matrix on a Large Dataset in Pandas?

I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' ...
Connor's user avatar
  • 651
2 votes
2 answers
632 views

What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?

To calculate the coefficient of partial determination R2 for a given variable: We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
skan's user avatar
  • 185
0 votes
1 answer
78 views

What is the fastest way to detect lag and calculate cross correlation of two binary time series?

Example, arr1 = array([0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0]) arr2 = array([1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0]) arr2 is almost perfectly correlated with ...
Imp's user avatar
  • 21
0 votes
0 answers
26 views

Which redundant feature should be use

I have two redundant features. A & B with 0.85 correlation. I know only one of them should be used to trained my model, but which feature should i use? A or B? Is there any method that can i use ...
Jovian Aditya's user avatar
2 votes
2 answers
834 views

Correlation vs Mutual Information vs let-the-model-decide

I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which ...
Álvaro V.'s user avatar
0 votes
1 answer
22 views

Different Correlation Coefficents with different Time Ranges

I built a Time-Series that displays the price of the Electricty Price in South Italy and two of their most important commodities (commodities, gas) used to produce the eletrical energy. So I ordered ...
giacomomaraglino's user avatar
0 votes
1 answer
75 views

Is it necessary to have a perfect correlation when using linear regression?

I am working on predicting BMI against weight, using linear regression. The scatter plot of the data can be found below. As you can see in the plot, there seems to be low (or no) correlation between ...
tip. rock's user avatar
0 votes
1 answer
66 views

R: the correlation is strong, but the graphic is very noisy

I have the following piece of code: ...
njhkugk6i76g6gi6gi7g6's user avatar
0 votes
0 answers
15 views

Find the most impactfuls parameters multivariate output unsupervised ML

I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
Art's user avatar
  • 1
1 vote
1 answer
234 views

Why Multicollinearity is a problem in machine learning algorithms

Is only a subset of algorithms are affected by the multicollinearity problem or all the machine learning algorithms? What is the solution for this?
Jagadeesh M's user avatar

1
2 3 4 5
9