Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
334
questions
0
votes
0answers
7 views
Does the Matthews correlation coefficient (MCC) make sense as a similarity measure between vectors?
I have a set of vectors representing items whose similarity I want to determine. Because the items are given as sets of features, the resulting one-hot encoding vectors are binary, so I am looking for ...
0
votes
0answers
9 views
How to determine correlation between ports and ticket prices?
I have titanic dataset, from which I have extracted ticket fares and embarkment ports.
I am trying to find out if there is a correlation between embarkment port and ticket fare, so I constructed ...
1
vote
0answers
7 views
R Phi Coefficient Calculation
I have a dataframe in R and am trying to determine the Phi correlation coefficient between 2 binary (aka dichotomous: 0 or 1) variables, each one in a column (column1 and column2).
I have installed ...
0
votes
0answers
13 views
Simple score function with 4 different indicators
I want to create a function, which returns a value between (0,1) or (-1,1). The result of this function is then used for a boolean decision. E.g. if the value is closer to 0 decision ...
0
votes
0answers
17 views
Correlation between continuous variables and multi class categorical variables in python
I was trying to figure out a way of finding a correlation between continuous variables and a non-binary target categorical label. The only thing I though of is by fitting the labels into Multinomial ...
0
votes
0answers
9 views
how to determine the relationship between attributes/whether one has impact the other
I am trying to build a model that determines whether two products have an impact on each other's sales performance. Ideally, the result will provide me a ranking/score between each 2 product pairs.
I ...
0
votes
0answers
10 views
How to find the lag that shows highest correlation between two time series variables?
I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
0
votes
0answers
30 views
How to measure (and get rid of) biases in training data?
There might be correlation in the training data that do not occur in 'nature' (or occur to a much smaller extent, or maybe we are legally required to not look at these correlations).
In my concrete ...
0
votes
1answer
12 views
Removing features that correlate with the target label
I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
0
votes
0answers
36 views
How to quantify the influence of multiple variables on a time series data
I have three variables in my time series data ($X_1,X_2,X_3$). I then calculate $y = (X_1+X_2)-X_3$ for each time $t$. Now, I need to look at a time interval and come up with a quantitative measure ...
0
votes
0answers
10 views
Analysing process data with sub groupings and checking for correlation
I have a dataset of process data for different equipment with many sensors. I would like to check the correlation of the different sensors to see if there is any strong correlation between some ...
1
vote
0answers
26 views
Finding correlations between plant features?
I have a list of flowers, and each has a color and a fragrance intensity. Each data row has a color and intensity for one breed of flower, like this:
...
0
votes
0answers
18 views
How to find correlations among repeated measures and how to analyze them?
I have a dataset of repeated measures and I would like to find any correlation between those measures. Also, after finding the correlation I would like to feed the dataset to an appropriate model and ...
3
votes
1answer
102 views
Finding if an outcome is predictable
Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
1
vote
1answer
17 views
If two features are highly correlated to each other and giving same impact on target variable in this case, which feature we need to select?
in my dataset, I am having plenty of features and two features are highly correlated to each other and giving same impact on target variable, in this case which feature we need to select in order to ...
1
vote
1answer
35 views
Why is my Tableau correlation coefficient different from what I calculated using Python?
I'm required to create a correlation matrix table using Tableau so I've created a version using Python to check if I did everything right.
The numbers are coming out different for both calculations. ...
2
votes
0answers
24 views
Why are correlation matrices used versus a matrix of R^2 values?
I'm relatively new to DS, so forgive me if this is a dumb question or in the wrong forum
When evaluating features it seems that almost everywhere a correlation matrix is used ...
2
votes
0answers
24 views
Cross correlation
I am trying to find a good algo (low latency) that is able to take two time series and determine which one is leading on the other one if any. The time series do not necessarily have the same ...
0
votes
3answers
52 views
Calculating optimal 'phase shift' when comparing two linear datasets to maximise correlation?
Let's say I have two columns of data, when graphed they look like this:
This does not have a particularly wonderful correlation.
On the other hand if I shift one graph 10 datapoints relative to the ...
1
vote
1answer
42 views
Meaning of the covariance matrix?
I wonder about the excessive usage of the covariance matrix across all kinds of machine learning tools. So far, for me, the covariance is just a pre-step to get to the correlation. And as there is an ...
1
vote
2answers
40 views
Correlation/Pattern Recognition in Lists
I am looking for algorithms to find pattern or more precise correlations in lists compared to an Output. Let us assume I have a Database like this:
Input: [A,C,D,E...], Output: Positive
Input: [A,B,C,...
1
vote
0answers
23 views
Should I remove features such as gender and birth month before drawing the heatmap because they are categorical?
I am working on a dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is categorical (1, 2, 3) and I am performing EDA on this dataset.
...
0
votes
0answers
11 views
N of Measurements needed for Multitrait Multimethod Matrix
I am taking a statistics subject for psychology and the content has recently covered multitrait multimethod matrices (MTMM). I am trying to wrap my head around it. Just as an example, for a matrix ...
0
votes
1answer
61 views
Association between continuous input and categorical output
I have two independent continuous variables like Age, Price and an outcome variable like purchased or not purchased
Now, which test should I use to ascertain the association of continuous input ...
0
votes
1answer
63 views
A feature highly correlated with target variable
What if one of the predictor variables is highly correlated with the target variable (say 0.9), what should we do? Should we drop it or keep it to build the prediction model(classification or ...
1
vote
0answers
32 views
I have a data set of optimal values after simulations, How can I find if this dataset follows a specific pattern or any relation exists?
In the simulation I am conducting, I have a set of triangles and I select the optimal triangle based on my metric. After every simulation, I obtain an optimal triangle and I note down the lengths of ...
2
votes
0answers
17 views
correlated variables & model performance: optimal trade-off
on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift.
assuming little to medium ...
1
vote
0answers
100 views
Pearson correlation on two categorical variables
I am using the fourth-corner method in one of my papers (for those who need the name). The method was developed to test associations between variables in two datasets. In my case, the datasets ...
0
votes
0answers
3 views
Timeserieses Xi are noisy, and the noise depends on a timeseries Y. How do I remove that noise, to smooth the timeserieses Xi?
The noises in my timeserieses Xi are highly correlated on the timeseries Y. How would you approach the problem of removing that noise? Ideally I would like all the Xi curves to be smooth, as well as ...
0
votes
0answers
8 views
Is there a difference between correlation rules and sequential patterns?
In security I read correlation rules in SIEM systems while in data mining I read sequential patterns. They appear to be something similar, because in both cases the order of itemsets matter. Is there ...
1
vote
1answer
19 views
Visualization methods in R to examine correlation of labels against response
Question
What are some good plotting methods in R for examining the relationship between a target variable and various explanatory variables? In particular, I'm looking for visualization techniques ...
0
votes
1answer
23 views
Evaluating correlations between dichotomous/binary variables? What test?
I have 4 variables, each are dichotomous, 1 = in this group & 0 = not in this group (for each of the 4 groups).
I have read mixed accounts of whether ...
0
votes
0answers
18 views
What is the difference between causal discovery and inverse modeling?
I do not see these words used interchangeably, but they seem to be similar. In inverse modeling we are trying to find causal factors given an effect. In causal discovery, we are also looking for ...
2
votes
0answers
13 views
Which method to use to remove correlation between independent variables comprising of both categorical and numerical variables? [duplicate]
The independent variables in the dataset contains categorical variables such as
Gender ( 2 levels)
Mode of Shipment ( 3 levels)
Product Importance ( 4 levels)
and Numerical Variables such as
...
0
votes
3answers
88 views
Removing correlation between independent variables
If there are two continuous independent variables that show a high amount of correlation between them, can we remove this correlation by multiplying or dividing the values of one of the variables with ...
1
vote
0answers
20 views
Analysis of changing correlation
Given a correlation matrix M and after an event, M changes.
What is the best way to analyse the change? Verify significance (presumably T-test, ANOVA..)
Currently looking at the %, count & average ...
1
vote
0answers
46 views
Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?
I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
0
votes
0answers
13 views
At what correlation factor we consider two features highly correlated so that it is safe to drop one feature for supervised learning?
At what correlation factor we consider two features highly correlated so that it is safe to drop one feature for supervised learning?
It is said that in supervised learning we should remove the high ...
1
vote
0answers
23 views
How to get correlation between the categories of two categorical variable?
I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
0
votes
0answers
11 views
Relative scoring of entities
I'm storing a set of house data in my Elasticsearch database with various attributes for each entity (such as price, number of bathrooms, sqft, etc..) I want to create a basic ranking engine that ...
0
votes
1answer
24 views
How to compute time-lagged correlation between two variables with many examples at each time t?
I have a dictionary of following form:
datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year
So, essentially, I'm trying to get an aggregate ...
0
votes
0answers
25 views
Find top features that determine movie rank
I am trying to analyze a movie dataset in order to find the specific features which determine whether or not the movie is in the top-10 movies of the year (or likewise the worst-10 movies of the year)....
1
vote
0answers
36 views
Should I drop Nan before before creating a scatter plot and computing the correlation matrix?
I have some NaN values in my data and I cannot replace it with median or something else.
Now I have to check out the correlation, using scatter plot and Pearson correlation.
Should I drop these NaNs ...
0
votes
1answer
14 views
Are there readily available models that can handle conditional correlation?
I've been working my way through the features of the Kaggle House Prices dataset (Note: this is a non-ranking entry, so this is just for exercises), and I've found a couple situations where there is a ...
0
votes
0answers
17 views
What are practical benefits of the various distance metrics in scipy?
I’m looking for a information about distance metrics.
Python’s scipy gives many different metrics, and I’d like to know more about their practical use.
In particular, I’m trying to understand the ...
1
vote
1answer
40 views
Heat map and correlation among variables [closed]
I would have a question on heat map and correlation among variables.
I created this heat map, looking at possible correlation among variables and target. I got very small values.
I wanted to set a ...
1
vote
1answer
26 views
Dimensionality reduction to correlate large number of variables
So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a ...
1
vote
0answers
38 views
What can I infer from a linear correlation of regression coefficients?
I am working on a dataset for classification where each observation is a series of values of a certain measurement $Y$ for a fixed range of values of measurement $X$ (i.e. a discrete mapping from $A \...
1
vote
0answers
21 views
Are convolutions in deep learning associative?
Let's denote "convolution in deep learing" as "convolution-deep", and "convolution in math or signal processing" as "convolution-math".
As we all know, ...
3
votes
1answer
30 views
Does a Convolutional Layer in a Neural Network learn the correlation between its input signals via its kernel?
I am interested in the theory behing what a convolutional neural network learns with its convolutional operations. I think it learns (useful) kernels which measure the correlation between its input ...
