# Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

401 questions
Filter by
Sorted by
Tagged with
489 views

### Changes in the standard Heatmap plot - symmetric bar colors, show only diagonal values, and column names at x,y axis ticks

I have a heatmap image (correlation between all matrix columns) and I'm straggling to preform all the changes below within the same image: bar colors should be symmetric around zero (e.g., ...
1 vote
52 views

### Which correlation test can be used on low count contigency table?

I need to find the correlation between the ratings given by two parties. Let's say we have 2 coaches, namely A and B. Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "...
70 views

### Pearson correlation with overlapping data

I have a financial time series and I want to calculate correlation between past and future returns. First I select look back and holding periods, say l and h respectively. Then I calculate past ...
697 views

### how to confirm a correlation between features

i have created a matrix to show the correlation between different features in this data. i do not believe my findings. surely the blp (price to buy the car), should correlate with the rental price ...
166 views

### How to interpret two continous variables output using GAM?

I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (...
1 vote
170 views

### Should I drop Nan before before creating a scatter plot and computing the correlation matrix?

I have some NaN values in my data and I cannot replace it with median or something else. Now I have to check out the correlation, using scatter plot and Pearson correlation. Should I drop these NaNs ...
1 vote
162 views

### Alternative methods for novelty detection and correlations

Hey mates I have the following project: Imagine having two datasets A and B. Each dataset consits of 101 time series with the ...
70 views

### Generate a set of values that has a given correlation with n given other sets of values

Given lists $L_1, \dots, L_n$ of, let's say, 2000 values each and arbitrary numbers $c_1, \dots, c_n$, is it possible to generate a random list of 2000 values that has correlation $c_i$ with $L_i$ for ...
155 views

### Is there any solution for modified t-test in python for spatial data?

I am searching for a solution to conduct a modified t-test or any other possibility in order to compute significant or non-significant relations in a spatial dataset (which can be a stack of rasters ...
148 views

### Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
26 views

### pattern within autocorrelation confidence intervals

I have a time-series producing the following auto & partial correlation plots. What insights can we make when there are oscillatory patterns within the grey region of insignificance? Does it make ...
1 vote
98 views

### Is it possible for a feature not correlated with a dependent variable to become important in a machine learning model?

Is it possible for a feature not correlated (or faintly correlated) with a dependent variable to become important in a machine learning model?
306 views

### Feature selection for time series prediction

I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables. When calculating correlation between variables (e.g. Close price of Tesla vs ...
145 views

### Scaling and handling highly correlated features in tabular data for regression

I am working on a regression problem trying to predict a target variable with seven predictor variables. I have a tabular dataset of 1400 rows. Before delving into the machine learning to build a ...
116 views

### Can we predict correlation between independent variables based on dependent variables

I have a correlation matrix for dependent var Vs independent variable as below: Year 1 Sales Sales -0.5453 1.0000 PriceIndex -0.6089 Income -0.5033 Interest -0.3842 As we ...
17 views

### How to check if an event affects time series

We have time series data. Depended variable – interest rates, about 15 years, monthly data. Independent variable – event, rating announcement (rating may change or may not), happens 2-3 times per year,...
38 views

### Correlation between predictions vs correlation between targets

In a multi-target model framework - where a separate model is estimated for each target - how can one take into account for correlations between targets during the training process ? For example say I ...
58 views

### Data preprocessing for Multiple Linear Regression Problem

For multiple linear regression problem, I have both categorical and numerical variables in the data. I am checking the correlation for numerical variables for EDA and standardizing them by taking log. ...
1 vote
62 views

### Finding the worst affected industry due to COVID in terms of unemployment

My goal is to find the worst affected industries from COVID—19 in terms unemployment. In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
190 views

### How can I take the cross-correlation between two discrete signal with slightly different discrete time points?

I have two discrete signals x1 and x2 with corresponding time points t1 and ...
376 views

### Relationship between two continuous variables in time series data

I have a dataset that collects daily data based on transactions between two entities. I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of ...
195 views

### How to find the correlation coefficient of one piece of data in a set?

I have a set of two pieces of data with a 0.85 coefficient. How can I calculate the coefficient for each individual set of data? For example, one set is 0-200 (x) and the other one is 0-\$500,000 (y)....
2k views

### corr() is giving an error. please help out of this problem and tell me what is this error about

when I am trying to run sns.heatmap(df.corr(),annot=True) this code in my jupyter notebook. this error is occuring. I cannot understand this problem. please help me.
1k views

### Python: calculate the weighted average correlation coefficient

I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
353 views

### How do we distinguish between correlated and un-correlated features/variables ? Is it relevant for a regression analysis?

Correlated and un-correlated terms are frequently used in data-science and understood as if they represent correlation coefficient. Is it the right way?
209 views

### Regression or Correlation for this RQ?

Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in ...
1k views

### SelectKBest and Correlation returns me excatly same feature selection. How?

Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But ...
1 vote
2k views

### Correlation Matrix Issues: Can't Convert Variables to Numerics

QUESTIONS: Can you tell me how I can convert int and factor variables to numeric variables in R? Can you tell me why ...
148 views

### How to correctly compute the correlation index of a column value from table in Python 3?

I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
83 views

### Correlation with target variable for regression problem

Given the following dataframe age job salary 0 1 Doctor 100 1 2 Engineer 200 2 3 Lawyer 300 ... with ...
1 vote
294 views

### Dimensionality reduction to correlate large number of variables

So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a ...
106 views

### Why does the order of columns in CCA change some results?

When running CCA from scikit learn, if the order is changed in the columns like (changing the order of the rows of both datasets together does not produce different results that I've seen -- not shown ...
166 views

### How to find correlations between data over time?

I have daily data of my sales and a daily update of how the COVID-19 cases increase. My daily sales contains information on my customer and on my product. My end goal would be to see if certain ...
347 views

### Similarity Measure of Simulated Time Series vs Observed time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
128 views

### How to compute time-lagged correlation between two variables with many examples at each time t?

I have a dictionary of following form: datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year So, essentially, I'm trying to get an aggregate ...
1 vote
119 views

### Handling Covariate Shift and Multi-collinearity in same Dataset

The problem is related to Regression problem. I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly ...
13 views

### How to visualize correlation values coming from two data frame for comparison

I am working on a project wherein we are comparing two methods used for modeling gene expression. One method is using elastic net and other is using lasso regression. In one method: we see that ...
106 views

### Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
19 views

### Ways to check whether your predictors are even correlated with their label?

I scraped a dataset of pre-match data in a video game and am trying to classify them by outcome, i.e. wins v. losses. The models I've tried so far have poor accuracy of around 50%, so I'm thinking ...
26 views

### Measuring features effect and importance in Partial Least Square (PLS) regression

Context: it is possible to assess features importance and effect for a model using model-independent scoring techniques such as Partial Dependence (PD) profile, Acculumated Local Effect (ALE) profile, ...
97 views

### Calculating correlation between embedding features

I am looking for a technique to calculate correlation for embedding features (array of floats). I'm interested in the correlation between features (embedding-embedding) as well as between feature and ...
11 views

### Correlation in forecast error

I am working on a time series project and while calculating the prediction intervals using bootstrapping method I have to fulfill the assumptions of it. It has an assumption of uncorrelated forecast ...
36 views

### TSNE plots of random data subsets are vastly different but labels are still clearly separated - what conclusions can we draw about the dataset?

I scraped a dataset of match data in a video game and labeled them according to their outcome (0 for loss, 1 for win). I wanted to see if there was actually any inherent relationship between the ...
1 vote
43 views

### How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
95 views

### Correlation Analysis dealing with missing value, when the missing values are actually expected

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here ...
10 views

### What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
17 views

### Good candidate time series models for 70-100 monthly data points, also incorporate past and future exogenous covariates

Per title, we're trying to identify a good time series modeling technique for: 70-100 variables of monthly sales or volume data (2015 or 2018 to present) Ability to forecast not only using trend and ...
1 vote
8k views

### Feature importance in neural networks

Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, ...