Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

120 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
0 answers
92 views

Fitting model to differenced time series

I have a time series on daily stock price of company(2013 data points).I took a first order difference and the following acf and pacf plots of the differenced series were obtained. However, I am ...
Jor_El's user avatar
  • 231
5 votes
2 answers
164 views

How to interpret two continous variables output using GAM?

I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (...
Hasan Sohail's user avatar
4 votes
1 answer
1k views

SelectKBest and Correlation returns me excatly same feature selection. How?

Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But ...
justRandomLearner's user avatar
3 votes
2 answers
337 views

Similarity Measure of Simulated Time Series vs Observed time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
Wiesel's user avatar
  • 51
3 votes
1 answer
306 views

Feature selection for time series prediction

I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables. When calculating correlation between variables (e.g. Close price of Tesla vs ...
Leandro Ercoli's user avatar
3 votes
1 answer
64 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
ImranAli's user avatar
2 votes
0 answers
36 views

Correlation between predictions vs correlation between targets

In a multi-target model framework - where a separate model is estimated for each target - how can one take into account for correlations between targets during the training process ? For example say I ...
Kreol's user avatar
  • 121
2 votes
0 answers
584 views

Library for Phi correlation coefficient in python?

I want to calculate correlation b/w categorical features in my data. I reviewed the literature and found phi coefficient can be used for this purpose. I found one library called phik in python enter ...
As13's user avatar
  • 77
2 votes
0 answers
31 views

Why are correlation matrices used versus a matrix of R^2 values?

I'm relatively new to DS, so forgive me if this is a dumb question or in the wrong forum When evaluating features it seems that almost everywhere a correlation matrix is used ...
Donovin's user avatar
  • 21
2 votes
0 answers
43 views

Cross correlation

I am trying to find a good algo (low latency) that is able to take two time series and determine which one is leading on the other one if any. The time series do not necessarily have the same ...
percojazz's user avatar
  • 121
2 votes
1 answer
153 views

Is there any solution for modified t-test in python for spatial data?

I am searching for a solution to conduct a modified t-test or any other possibility in order to compute significant or non-significant relations in a spatial dataset (which can be a stack of rasters ...
dh81's user avatar
  • 21
2 votes
0 answers
24 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
JungleDiff's user avatar
2 votes
0 answers
221 views

Similarity Measure Time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
Wiesel's user avatar
  • 51
2 votes
0 answers
117 views

Problems with Graphical Lasso

I'm trying to use the Graphical Lasso algorithm (more specifically the R package glasso) to find an estimated graph representing the connections between a set of nodes by estimating a precision matrix....
F Lourenço's user avatar
2 votes
0 answers
104 views

Type of Test to Determine Correlation in R

I have a dataset of approximately 48,000 rows each one a click of a an article, some of these clicks were also comments. For each article I have the country and subject of the article and name of ...
Jonathan Dine's user avatar
2 votes
1 answer
115 views

Can we predict correlation between independent variables based on dependent variables

I have a correlation matrix for dependent var Vs independent variable as below: Year 1 Sales Sales -0.5453 1.0000 PriceIndex -0.6089 Income -0.5033 Interest -0.3842 As we ...
Shivi's user avatar
  • 31
1 vote
0 answers
20 views

How to build a model with variables whose effects might be delayed?

As a personal project, I want to see how certain lifestyle choices (such as alcohol consumption, exercise time, hours worked/slept, time I get up and go to sleep) affect my mood. While I know the ...
Dave Moutardier's user avatar
1 vote
1 answer
119 views

Handling Covariate Shift and Multi-collinearity in same Dataset

The problem is related to Regression problem. I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly ...
Debadri Dutta's user avatar
1 vote
1 answer
93 views

Is it possible for a feature not correlated with a dependent variable to become important in a machine learning model?

Is it possible for a feature not correlated (or faintly correlated) with a dependent variable to become important in a machine learning model?
NAS_2339's user avatar
  • 243
1 vote
0 answers
530 views

Is pearson correlation matrix a good indicator for label encoded categorical and numeric independent data?

I have a dataset having 22 independent variables out of which 15 are categorical data that has already been label encoded i.e the dtype is int64 and the contents are in a range of 0 to n (n is the ...
Echo's user avatar
  • 113
1 vote
0 answers
22 views

Approach for finding patterns in daily event data

I am software engineer but have zero data science background, so apologies for this basic question. I would like to find correlations between different behaviors of my daily life and different ...
driangle's user avatar
  • 111
1 vote
0 answers
67 views

statistical tests for null hypothesis - what if model is non linear?

I am reading the "An Introduction to Statistical Learning" (Gareth James & alii, Springer) as a primer to machine learning. I am reading the part in linear regressors, and learnt there ...
user305883's user avatar
1 vote
0 answers
62 views

Feature creation: Problem with correlated features?

I recently started to read about feature creation. I've seen some general guidelines although I am not really sure if they are completely true, for example: 1 - Linear classifiers for binary ...
embedded_dev's user avatar
1 vote
1 answer
46 views

Find feature categories associated with a specific target category

I have a Dataset with three columns. Products (up to 200). The quality checks that have not been conducted at the final quality check. (Up to 70 different Quality Control Measures) The result of the ...
can.inan's user avatar
1 vote
1 answer
27 views

How to test whether data is clustered wrt. subcategories?

I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
user avatar
1 vote
0 answers
31 views

Finding correlations between plant features?

I have a list of flowers, and each has a color and a fragrance intensity. Each data row has a color and intensity for one breed of flower, like this: ...
Nicholas Martin's user avatar
1 vote
0 answers
42 views

Should I remove features such as gender and birth month before drawing the heatmap because they are categorical?

I am working on a dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is categorical (1, 2, 3) and I am performing EDA on this dataset. ...
leahnanno's user avatar
1 vote
0 answers
33 views

I have a data set of optimal values after simulations, How can I find if this dataset follows a specific pattern or any relation exists?

In the simulation I am conducting, I have a set of triangles and I select the optimal triangle based on my metric. After every simulation, I obtain an optimal triangle and I note down the lengths of ...
Sohaib Bin Altaf's user avatar
1 vote
0 answers
25 views

correlated variables & model performance: optimal trade-off

on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift. assuming little to medium ...
Asher11's user avatar
  • 159
1 vote
0 answers
769 views

Pearson correlation on two categorical variables

I am using the fourth-corner method in one of my papers (for those who need the name). The method was developed to test associations between variables in two datasets. In my case, the datasets ...
Pierre O's user avatar
1 vote
0 answers
23 views

Analysis of changing correlation

Given a correlation matrix M and after an event, M changes. What is the best way to analyse the change? Verify significance (presumably T-test, ANOVA..) Currently looking at the %, count & average ...
Teabelly's user avatar
  • 111
1 vote
0 answers
106 views

Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
Prospero's user avatar
1 vote
1 answer
42 views

How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
bonaqua's user avatar
  • 11
1 vote
1 answer
280 views

Dimensionality reduction to correlate large number of variables

So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a ...
Masterloop's user avatar
1 vote
0 answers
42 views

What can I infer from a linear correlation of regression coefficients?

I am working on a dataset for classification where each observation is a series of values of a certain measurement $Y$ for a fixed range of values of measurement $X$ (i.e. a discrete mapping from $A \...
Anakhand's user avatar
  • 111
1 vote
0 answers
53 views

Are convolutions in deep learning associative?

Let's denote "convolution in deep learing" as "convolution-deep", and "convolution in math or signal processing" as "convolution-math". As we all know, ...
WBR's user avatar
  • 21
1 vote
0 answers
73 views

High correlation between the independent and dependent variables but low performance of regression model

I have a dataset of 4900 rows and 2060 feature. I calculated the correlation using kendall method between the dependent and independent features, and found out that 5 of these features are having a ...
Mee's user avatar
  • 121
1 vote
0 answers
77 views

The Merits of Feature Reduction Routines

I am interested in learning what routine others use (if any) for Feature Reduction/Selection. For example, If my data has several thousand features, I typically try {2,3,4} things right away depending ...
mccurcio's user avatar
  • 223
1 vote
1 answer
62 views

Finding the worst affected industry due to COVID in terms of unemployment

My goal is to find the worst affected industries from COVID—19 in terms unemployment. In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
NAS_2339's user avatar
  • 243
1 vote
0 answers
77 views

Pairwise 3D object correlation between 2 objects

I have a dataset which contains 3D CT scans from different patients along with the segmenation masks of a certain organ. The 3D scans have been drawn each day for a period of 30 days for each patient. ...
Dimimal13's user avatar
1 vote
0 answers
30 views

lags number in multivariate time series analysis correlation

I am trying to calculate the correlation coefficient(Pearson's r) of a financial time series $Y(t)$ and other exogenous variables $X_1(t),..., X_n(t)$. I am trying to understand the impact of my ...
BalticOY's user avatar
1 vote
1 answer
41 views

Get an equation from the relationship among variables

I have an agricultural dataset of crop yield (in kilograms) under environmental conditions such as humidity, rainfall, sunlight over few years. I want to find out relationship between the ...
nipunasudha's user avatar
1 vote
0 answers
108 views

Scale (frequency) invariant cross-correlation

I am looking for cross-correlation of two 'signals' that would be invariant of scale(frequency, or sampling). To give you example what I mean: Standard cross-correlation of sin(x) and cos(x) find ...
Martin's user avatar
  • 111
1 vote
0 answers
126 views

Plotting heatmap seaborn dilemma

First i want to tell you that i'm trying to create a model in the right steps, which needs data exploration, data preprocessing (AFTER splitting the data into train/test), etc. For the part of data ...
random student's user avatar
1 vote
3 answers
50 views

Which correlation test can be used on low count contigency table?

I need to find the correlation between the ratings given by two parties. Let's say we have 2 coaches, namely A and B. Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "...
Johnny's user avatar
  • 11
1 vote
0 answers
108 views

Effect of a single variable in a linear model

In a multiple linear regression (MLR) model, people normally look at the relationship between each predictor and the response variable to see if it is linear. However, in my example below, I show that ...
towi_parallelism's user avatar
1 vote
0 answers
49 views

Select the right distribution

I have a dataset like: ...
Nathalie's user avatar
  • 147
1 vote
1 answer
2k views

Correlation Matrix Issues: Can't Convert Variables to Numerics

QUESTIONS: Can you tell me how I can convert int and factor variables to numeric variables in R? Can you tell me why ...
Andrew Slutzky's user avatar
1 vote
0 answers
1k views

Pearson's Coefficient for classification problem

Can we use Pearson's Correlation Coefficient to calculate the correlation between features and target in the case of a classification problem?
Anju's user avatar
  • 23
1 vote
0 answers
334 views

Spearmanr on two pandas dataframes

I have two pandas dataframes that look about the same but with different information stored in them. My question will be about how to compare the two dataframes to ensure column and row match before ...
Stephopolis's user avatar