Skip to main content

Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
0 votes
1 answer
489 views

Changes in the standard Heatmap plot - symmetric bar colors, show only diagonal values, and column names at x,y axis ticks

I have a heatmap image (correlation between all matrix columns) and I'm straggling to preform all the changes below within the same image: bar colors should be symmetric around zero (e.g., ...
1 vote
3 answers
52 views

Which correlation test can be used on low count contigency table?

I need to find the correlation between the ratings given by two parties. Let's say we have 2 coaches, namely A and B. Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "...
0 votes
1 answer
70 views

Pearson correlation with overlapping data

I have a financial time series and I want to calculate correlation between past and future returns. First I select look back and holding periods, say l and h respectively. Then I calculate past ...
0 votes
2 answers
697 views

how to confirm a correlation between features

i have created a matrix to show the correlation between different features in this data. i do not believe my findings. surely the blp (price to buy the car), should correlate with the rental price ...
5 votes
2 answers
166 views

How to interpret two continous variables output using GAM?

I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (...
1 vote
1 answer
170 views

Should I drop Nan before before creating a scatter plot and computing the correlation matrix?

I have some NaN values in my data and I cannot replace it with median or something else. Now I have to check out the correlation, using scatter plot and Pearson correlation. Should I drop these NaNs ...
1 vote
2 answers
162 views

Alternative methods for novelty detection and correlations

Hey mates I have the following project: Imagine having two datasets A and B. Each dataset consits of 101 time series with the ...
0 votes
1 answer
70 views

Generate a set of values that has a given correlation with n given other sets of values

Given lists $L_1, \dots, L_n$ of, let's say, 2000 values each and arbitrary numbers $c_1, \dots, c_n$, is it possible to generate a random list of 2000 values that has correlation $c_i$ with $L_i$ for ...
2 votes
1 answer
155 views

Is there any solution for modified t-test in python for spatial data?

I am searching for a solution to conduct a modified t-test or any other possibility in order to compute significant or non-significant relations in a spatial dataset (which can be a stack of rasters ...
0 votes
2 answers
148 views

Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
0 votes
0 answers
26 views

pattern within autocorrelation confidence intervals

I have a time-series producing the following auto & partial correlation plots. What insights can we make when there are oscillatory patterns within the grey region of insignificance? Does it make ...
1 vote
1 answer
98 views

Is it possible for a feature not correlated with a dependent variable to become important in a machine learning model?

Is it possible for a feature not correlated (or faintly correlated) with a dependent variable to become important in a machine learning model?
3 votes
1 answer
306 views

Feature selection for time series prediction

I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables. When calculating correlation between variables (e.g. Close price of Tesla vs ...
0 votes
1 answer
145 views

Scaling and handling highly correlated features in tabular data for regression

I am working on a regression problem trying to predict a target variable with seven predictor variables. I have a tabular dataset of 1400 rows. Before delving into the machine learning to build a ...
2 votes
1 answer
116 views

Can we predict correlation between independent variables based on dependent variables

I have a correlation matrix for dependent var Vs independent variable as below: Year 1 Sales Sales -0.5453 1.0000 PriceIndex -0.6089 Income -0.5033 Interest -0.3842 As we ...
0 votes
1 answer
17 views

How to check if an event affects time series

We have time series data. Depended variable – interest rates, about 15 years, monthly data. Independent variable – event, rating announcement (rating may change or may not), happens 2-3 times per year,...
2 votes
0 answers
38 views

Correlation between predictions vs correlation between targets

In a multi-target model framework - where a separate model is estimated for each target - how can one take into account for correlations between targets during the training process ? For example say I ...
0 votes
1 answer
58 views

Data preprocessing for Multiple Linear Regression Problem

For multiple linear regression problem, I have both categorical and numerical variables in the data. I am checking the correlation for numerical variables for EDA and standardizing them by taking log. ...
1 vote
1 answer
62 views

Finding the worst affected industry due to COVID in terms of unemployment

My goal is to find the worst affected industries from COVID—19 in terms unemployment. In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
0 votes
1 answer
190 views

How can I take the cross-correlation between two discrete signal with slightly different discrete time points?

I have two discrete signals x1 and x2 with corresponding time points t1 and ...
0 votes
1 answer
376 views

Relationship between two continuous variables in time series data

I have a dataset that collects daily data based on transactions between two entities. I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of ...
0 votes
1 answer
195 views

How to find the correlation coefficient of one piece of data in a set?

I have a set of two pieces of data with a 0.85 coefficient. How can I calculate the coefficient for each individual set of data? For example, one set is 0-200 (x) and the other one is 0-$500,000 (y)....
0 votes
2 answers
2k views

corr() is giving an error. please help out of this problem and tell me what is this error about

when I am trying to run sns.heatmap(df.corr(),annot=True) this code in my jupyter notebook. this error is occuring. I cannot understand this problem. please help me.
0 votes
1 answer
1k views

Python: calculate the weighted average correlation coefficient

I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
0 votes
2 answers
353 views

How do we distinguish between correlated and un-correlated features/variables ? Is it relevant for a regression analysis?

Correlated and un-correlated terms are frequently used in data-science and understood as if they represent correlation coefficient. Is it the right way?
2 votes
2 answers
209 views

Regression or Correlation for this RQ?

Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in ...
4 votes
1 answer
1k views

SelectKBest and Correlation returns me excatly same feature selection. How?

Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But ...
1 vote
1 answer
2k views

Correlation Matrix Issues: Can't Convert Variables to Numerics

QUESTIONS: Can you tell me how I can convert int and factor variables to numeric variables in R? Can you tell me why ...
0 votes
1 answer
148 views

How to correctly compute the correlation index of a column value from table in Python 3?

I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
0 votes
1 answer
83 views

Correlation with target variable for regression problem

Given the following dataframe age job salary 0 1 Doctor 100 1 2 Engineer 200 2 3 Lawyer 300 ... with ...
1 vote
1 answer
294 views

Dimensionality reduction to correlate large number of variables

So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a ...
0 votes
1 answer
106 views

Why does the order of columns in CCA change some results?

When running CCA from scikit learn, if the order is changed in the columns like (changing the order of the rows of both datasets together does not produce different results that I've seen -- not shown ...
0 votes
3 answers
166 views

How to find correlations between data over time?

I have daily data of my sales and a daily update of how the COVID-19 cases increase. My daily sales contains information on my customer and on my product. My end goal would be to see if certain ...
3 votes
2 answers
347 views

Similarity Measure of Simulated Time Series vs Observed time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
0 votes
1 answer
128 views

How to compute time-lagged correlation between two variables with many examples at each time t?

I have a dictionary of following form: datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year So, essentially, I'm trying to get an aggregate ...
1 vote
1 answer
119 views

Handling Covariate Shift and Multi-collinearity in same Dataset

The problem is related to Regression problem. I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly ...
0 votes
0 answers
13 views

How to visualize correlation values coming from two data frame for comparison

I am working on a project wherein we are comparing two methods used for modeling gene expression. One method is using elastic net and other is using lasso regression. In one method: we see that ...
5 votes
2 answers
106 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
0 votes
0 answers
19 views

Ways to check whether your predictors are even correlated with their label?

I scraped a dataset of pre-match data in a video game and am trying to classify them by outcome, i.e. wins v. losses. The models I've tried so far have poor accuracy of around 50%, so I'm thinking ...
0 votes
0 answers
26 views

Measuring features effect and importance in Partial Least Square (PLS) regression

Context: it is possible to assess features importance and effect for a model using model-independent scoring techniques such as Partial Dependence (PD) profile, Acculumated Local Effect (ALE) profile, ...
0 votes
0 answers
97 views

Calculating correlation between embedding features

I am looking for a technique to calculate correlation for embedding features (array of floats). I'm interested in the correlation between features (embedding-embedding) as well as between feature and ...
0 votes
0 answers
11 views

Correlation in forecast error

I am working on a time series project and while calculating the prediction intervals using bootstrapping method I have to fulfill the assumptions of it. It has an assumption of uncorrelated forecast ...
0 votes
1 answer
36 views

TSNE plots of random data subsets are vastly different but labels are still clearly separated - what conclusions can we draw about the dataset?

I scraped a dataset of match data in a video game and labeled them according to their outcome (0 for loss, 1 for win). I wanted to see if there was actually any inherent relationship between the ...
1 vote
1 answer
43 views

How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
0 votes
2 answers
95 views

Correlation Analysis dealing with missing value, when the missing values are actually expected

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here ...
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
0 votes
0 answers
17 views

Good candidate time series models for 70-100 monthly data points, also incorporate past and future exogenous covariates

Per title, we're trying to identify a good time series modeling technique for: 70-100 variables of monthly sales or volume data (2015 or 2018 to present) Ability to forecast not only using trend and ...
1 vote
4 answers
8k views

Feature importance in neural networks

Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, ...
0 votes
0 answers
63 views

Correlation in R between categorical and boolean variables

I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
0 votes
0 answers
40 views

What is reliability? How it is related to correlation coefficient?

We can compute real or population correlation(rho) by square-root of 1 minus R-squared.Is this a correct interpretation? Does population correlation mean a real correlation measured as square-root of ...

1
2 3 4 5
9