# Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

334 questions
Filter by
Sorted by
Tagged with
7 views

### Does the Matthews correlation coefficient (MCC) make sense as a similarity measure between vectors?

I have a set of vectors representing items whose similarity I want to determine. Because the items are given as sets of features, the resulting one-hot encoding vectors are binary, so I am looking for ...
9 views

### How to determine correlation between ports and ticket prices?

I have titanic dataset, from which I have extracted ticket fares and embarkment ports. I am trying to find out if there is a correlation between embarkment port and ticket fare, so I constructed ...
7 views

### R Phi Coefficient Calculation

I have a dataframe in R and am trying to determine the Phi correlation coefficient between 2 binary (aka dichotomous: 0 or 1) variables, each one in a column (column1 and column2). I have installed ...
13 views

### Simple score function with 4 different indicators

I want to create a function, which returns a value between (0,1) or (-1,1). The result of this function is then used for a boolean decision. E.g. if the value is closer to 0 decision ...
17 views

### Correlation between continuous variables and multi class categorical variables in python

I was trying to figure out a way of finding a correlation between continuous variables and a non-binary target categorical label. The only thing I though of is by fitting the labels into Multinomial ...
9 views

### how to determine the relationship between attributes/whether one has impact the other

I am trying to build a model that determines whether two products have an impact on each other's sales performance. Ideally, the result will provide me a ranking/score between each 2 product pairs. I ...
10 views

### How to find the lag that shows highest correlation between two time series variables?

I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
30 views

### How to measure (and get rid of) biases in training data?

There might be correlation in the training data that do not occur in 'nature' (or occur to a much smaller extent, or maybe we are legally required to not look at these correlations). In my concrete ...
12 views

### Removing features that correlate with the target label

I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
36 views

### How to quantify the influence of multiple variables on a time series data

I have three variables in my time series data ($X_1,X_2,X_3$). I then calculate $y = (X_1+X_2)-X_3$ for each time $t$. Now, I need to look at a time interval and come up with a quantitative measure ...
10 views

### Analysing process data with sub groupings and checking for correlation

I have a dataset of process data for different equipment with many sensors. I would like to check the correlation of the different sensors to see if there is any strong correlation between some ...
26 views

### Finding correlations between plant features?

I have a list of flowers, and each has a color and a fragrance intensity. Each data row has a color and intensity for one breed of flower, like this: ...
18 views

### How to find correlations among repeated measures and how to analyze them?

I have a dataset of repeated measures and I would like to find any correlation between those measures. Also, after finding the correlation I would like to feed the dataset to an appropriate model and ...
102 views

### Finding if an outcome is predictable

Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
17 views

### If two features are highly correlated to each other and giving same impact on target variable in this case, which feature we need to select?

in my dataset, I am having plenty of features and two features are highly correlated to each other and giving same impact on target variable, in this case which feature we need to select in order to ...
35 views

### Why is my Tableau correlation coefficient different from what I calculated using Python?

I'm required to create a correlation matrix table using Tableau so I've created a version using Python to check if I did everything right. The numbers are coming out different for both calculations. ...
24 views

### Why are correlation matrices used versus a matrix of R^2 values?

I'm relatively new to DS, so forgive me if this is a dumb question or in the wrong forum When evaluating features it seems that almost everywhere a correlation matrix is used ...
24 views

### Cross correlation

I am trying to find a good algo (low latency) that is able to take two time series and determine which one is leading on the other one if any. The time series do not necessarily have the same ...
52 views

### Calculating optimal 'phase shift' when comparing two linear datasets to maximise correlation?

Let's say I have two columns of data, when graphed they look like this: This does not have a particularly wonderful correlation. On the other hand if I shift one graph 10 datapoints relative to the ...
42 views

### Meaning of the covariance matrix?

I wonder about the excessive usage of the covariance matrix across all kinds of machine learning tools. So far, for me, the covariance is just a pre-step to get to the correlation. And as there is an ...
40 views

### Correlation/Pattern Recognition in Lists

I am looking for algorithms to find pattern or more precise correlations in lists compared to an Output. Let us assume I have a Database like this: Input: [A,C,D,E...], Output: Positive Input: [A,B,C,...
23 views

### Should I remove features such as gender and birth month before drawing the heatmap because they are categorical?

I am working on a dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is categorical (1, 2, 3) and I am performing EDA on this dataset. ...
11 views

### N of Measurements needed for Multitrait Multimethod Matrix

I am taking a statistics subject for psychology and the content has recently covered multitrait multimethod matrices (MTMM). I am trying to wrap my head around it. Just as an example, for a matrix ...
61 views

### Association between continuous input and categorical output

I have two independent continuous variables like Age, Price and an outcome variable like purchased or not purchased Now, which test should I use to ascertain the association of continuous input ...
63 views

### A feature highly correlated with target variable

What if one of the predictor variables is highly correlated with the target variable (say 0.9), what should we do? Should we drop it or keep it to build the prediction model(classification or ...
32 views

### I have a data set of optimal values after simulations, How can I find if this dataset follows a specific pattern or any relation exists?

In the simulation I am conducting, I have a set of triangles and I select the optimal triangle based on my metric. After every simulation, I obtain an optimal triangle and I note down the lengths of ...
17 views

### correlated variables & model performance: optimal trade-off

on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift. assuming little to medium ...
100 views

### Pearson correlation on two categorical variables

I am using the fourth-corner method in one of my papers (for those who need the name). The method was developed to test associations between variables in two datasets. In my case, the datasets ...
3 views

### Timeserieses Xi are noisy, and the noise depends on a timeseries Y. How do I remove that noise, to smooth the timeserieses Xi?

The noises in my timeserieses Xi are highly correlated on the timeseries Y. How would you approach the problem of removing that noise? Ideally I would like all the Xi curves to be smooth, as well as ...
8 views

### Is there a difference between correlation rules and sequential patterns?

In security I read correlation rules in SIEM systems while in data mining I read sequential patterns. They appear to be something similar, because in both cases the order of itemsets matter. Is there ...
19 views

### Visualization methods in R to examine correlation of labels against response

Question What are some good plotting methods in R for examining the relationship between a target variable and various explanatory variables? In particular, I'm looking for visualization techniques ...
23 views

### Evaluating correlations between dichotomous/binary variables? What test?

I have 4 variables, each are dichotomous, 1 = in this group & 0 = not in this group (for each of the 4 groups). I have read mixed accounts of whether ...
18 views

### What is the difference between causal discovery and inverse modeling?

I do not see these words used interchangeably, but they seem to be similar. In inverse modeling we are trying to find causal factors given an effect. In causal discovery, we are also looking for ...
13 views

### Which method to use to remove correlation between independent variables comprising of both categorical and numerical variables? [duplicate]

The independent variables in the dataset contains categorical variables such as Gender ( 2 levels) Mode of Shipment ( 3 levels) Product Importance ( 4 levels) and Numerical Variables such as ...
88 views

### Removing correlation between independent variables

If there are two continuous independent variables that show a high amount of correlation between them, can we remove this correlation by multiplying or dividing the values of one of the variables with ...
20 views

### Analysis of changing correlation

Given a correlation matrix M and after an event, M changes. What is the best way to analyse the change? Verify significance (presumably T-test, ANOVA..) Currently looking at the %, count & average ...
46 views

### Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
13 views

### At what correlation factor we consider two features highly correlated so that it is safe to drop one feature for supervised learning?

At what correlation factor we consider two features highly correlated so that it is safe to drop one feature for supervised learning? It is said that in supervised learning we should remove the high ...
23 views

### How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
11 views

### Relative scoring of entities

I'm storing a set of house data in my Elasticsearch database with various attributes for each entity (such as price, number of bathrooms, sqft, etc..) I want to create a basic ranking engine that ...
24 views

### How to compute time-lagged correlation between two variables with many examples at each time t?

I have a dictionary of following form: datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year So, essentially, I'm trying to get an aggregate ...
25 views

### Find top features that determine movie rank

I am trying to analyze a movie dataset in order to find the specific features which determine whether or not the movie is in the top-10 movies of the year (or likewise the worst-10 movies of the year)....
36 views

### Should I drop Nan before before creating a scatter plot and computing the correlation matrix?

I have some NaN values in my data and I cannot replace it with median or something else. Now I have to check out the correlation, using scatter plot and Pearson correlation. Should I drop these NaNs ...
14 views

### Are there readily available models that can handle conditional correlation?

I've been working my way through the features of the Kaggle House Prices dataset (Note: this is a non-ranking entry, so this is just for exercises), and I've found a couple situations where there is a ...
17 views

### What are practical benefits of the various distance metrics in scipy?

I’m looking for a information about distance metrics. Python’s scipy gives many different metrics, and I’d like to know more about their practical use. In particular, I’m trying to understand the ...
40 views

### Heat map and correlation among variables [closed]

I would have a question on heat map and correlation among variables. I created this heat map, looking at possible correlation among variables and target. I got very small values. I wanted to set a ...
26 views

### Dimensionality reduction to correlate large number of variables

So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a ...
38 views

### What can I infer from a linear correlation of regression coefficients?

I am working on a dataset for classification where each observation is a series of values of a certain measurement $Y$ for a fixed range of values of measurement $X$ (i.e. a discrete mapping from \$A \...