Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
344
questions
1
vote
0answers
14 views
Approach for finding patterns in daily event data
I am software engineer but have zero data science background, so apologies for this basic question.
I would like to find correlations between different behaviors of my daily life and different ...
0
votes
0answers
21 views
what are the effect on machine learning regression model if the dataset has two exact same columns
What will be the effect on the Machine learning model if the dataset has two exact same columns(exact 1 correlation).
One thing that comes to my mind is that if two columns are exactly the same then ...
0
votes
0answers
17 views
What correlation is considered to be big for linear regression predictors?
It is well known that if two linear regression predictors highly correlate, it is bad for our model, but which correlation is considered to be big? Is it 0.5,0.6,0.8,0.9..? I have tried to find out ...
0
votes
1answer
18 views
Analysis with categorical variable
My dataset consists of a numeric variable (called "N4") and several categorical variables that affect the numeric variable. For example there is a categorical variable called "die" ...
1
vote
0answers
34 views
statistical tests for null hypothesis - what if model is non linear?
I am reading the "An Introduction to Statistical Learning" (Gareth James & alii, Springer) as a primer to machine learning.
I am reading the part in linear regressors, and learnt there ...
0
votes
0answers
14 views
Best method for calculating all correlations in a dataset?
First of all, to be clear, I don't want all correlations in the dataset, I just want to find all high correlations between the response variable and the features.
I posted this question to reddit on '...
1
vote
0answers
14 views
Feature creation: Problem with correlated features?
I recently started to read about feature creation. I've seen some general guidelines although I am not really sure if they are completely true, for example:
1 - Linear classifiers for binary ...
1
vote
1answer
27 views
Why is pearson correlation popular if it detects only linear correlation?
Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
1
vote
1answer
43 views
Find feature categories associated with a specific target category
I have a Dataset with three columns.
Products (up to 200).
The quality checks that have not been conducted at the final quality check. (Up to 70 different Quality Control Measures)
The result of the ...
0
votes
0answers
52 views
Predict multiple labels when labels are of mixed type: floating point and categorical
I was provided with a data file which looks like this:
...
0
votes
0answers
20 views
Is there a "tree-based-correlation" for tree-based algorithms?
Although correlated features are not a big issue when training tree-based models, they spoil model explainability. When several features correlate, sometimes they may be picked at random. Then their ...
1
vote
1answer
18 views
How to test whether data is clustered wrt. subcategories?
I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
0
votes
0answers
11 views
Event correlation in processes
I have a dataset with roughly 100 events per product (150 products). So each product has these 100 events. The last event is a quality check which is either red or green and which is either due or on ...
0
votes
2answers
39 views
If I have two variables with strong correlation, should I delete one and leave the other in my data
I have a large dataset, where I should make a binary prediction. The fact is that, after analyzing the data, I found that some variables are positively correlated to each other. So, I was wondering ...
0
votes
0answers
13 views
Does the Matthews correlation coefficient (MCC) make sense as a similarity measure between vectors?
I have a set of vectors representing items whose similarity I want to determine. Because the items are given as sets of features, the resulting one-hot encoding vectors are binary, so I am looking for ...
0
votes
0answers
10 views
How to determine correlation between ports and ticket prices?
I have titanic dataset, from which I have extracted ticket fares and embarkment ports.
I am trying to find out if there is a correlation between embarkment port and ticket fare, so I constructed ...
1
vote
0answers
14 views
R Phi Coefficient Calculation [closed]
I have a dataframe in R and am trying to determine the Phi correlation coefficient between 2 binary (aka dichotomous: 0 or 1) variables, each one in a column (column1 and column2).
I have installed ...
0
votes
0answers
15 views
Simple score function with 4 different indicators
I want to create a function, which returns a value between (0,1) or (-1,1). The result of this function is then used for a boolean decision. E.g. if the value is closer to 0 decision ...
0
votes
0answers
171 views
Correlation between continuous variables and multi class categorical variables in python
I was trying to figure out a way of finding a correlation between continuous variables and a non-binary target categorical label. The only thing I though of is by fitting the labels into Multinomial ...
0
votes
0answers
9 views
how to determine the relationship between attributes/whether one has impact the other
I am trying to build a model that determines whether two products have an impact on each other's sales performance. Ideally, the result will provide me a ranking/score between each 2 product pairs.
I ...
0
votes
0answers
14 views
How to find the lag that shows highest correlation between two time series variables?
I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
0
votes
0answers
30 views
How to measure (and get rid of) biases in training data?
There might be correlation in the training data that do not occur in 'nature' (or occur to a much smaller extent, or maybe we are legally required to not look at these correlations).
In my concrete ...
1
vote
1answer
24 views
Removing features that correlate with the target label
I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
0
votes
0answers
38 views
How to quantify the influence of multiple variables on a time series data
I have three variables in my time series data ($X_1,X_2,X_3$). I then calculate $y = (X_1+X_2)-X_3$ for each time $t$. Now, I need to look at a time interval and come up with a quantitative measure ...
0
votes
0answers
10 views
Analysing process data with sub groupings and checking for correlation
I have a dataset of process data for different equipment with many sensors. I would like to check the correlation of the different sensors to see if there is any strong correlation between some ...
1
vote
0answers
29 views
Finding correlations between plant features?
I have a list of flowers, and each has a color and a fragrance intensity. Each data row has a color and intensity for one breed of flower, like this:
...
0
votes
0answers
19 views
How to find correlations among repeated measures and how to analyze them?
I have a dataset of repeated measures and I would like to find any correlation between those measures. Also, after finding the correlation I would like to feed the dataset to an appropriate model and ...
3
votes
1answer
106 views
Finding if an outcome is predictable
Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
1
vote
1answer
19 views
If two features are highly correlated to each other and giving same impact on target variable in this case, which feature we need to select?
in my dataset, I am having plenty of features and two features are highly correlated to each other and giving same impact on target variable, in this case which feature we need to select in order to ...
1
vote
1answer
90 views
Why is my Tableau correlation coefficient different from what I calculated using Python?
I'm required to create a correlation matrix table using Tableau so I've created a version using Python to check if I did everything right.
The numbers are coming out different for both calculations. ...
2
votes
0answers
25 views
Why are correlation matrices used versus a matrix of R^2 values?
I'm relatively new to DS, so forgive me if this is a dumb question or in the wrong forum
When evaluating features it seems that almost everywhere a correlation matrix is used ...
2
votes
0answers
25 views
Cross correlation
I am trying to find a good algo (low latency) that is able to take two time series and determine which one is leading on the other one if any. The time series do not necessarily have the same ...
0
votes
3answers
67 views
Calculating optimal 'phase shift' when comparing two linear datasets to maximise correlation?
Let's say I have two columns of data, when graphed they look like this:
This does not have a particularly wonderful correlation.
On the other hand if I shift one graph 10 datapoints relative to the ...
1
vote
1answer
44 views
Meaning of the covariance matrix?
I wonder about the excessive usage of the covariance matrix across all kinds of machine learning tools. So far, for me, the covariance is just a pre-step to get to the correlation. And as there is an ...
1
vote
2answers
45 views
Correlation/Pattern Recognition in Lists
I am looking for algorithms to find pattern or more precise correlations in lists compared to an Output. Let us assume I have a Database like this:
Input: [A,C,D,E...], Output: Positive
Input: [A,B,C,...
1
vote
0answers
24 views
Should I remove features such as gender and birth month before drawing the heatmap because they are categorical?
I am working on a dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is categorical (1, 2, 3) and I am performing EDA on this dataset.
...
0
votes
0answers
11 views
N of Measurements needed for Multitrait Multimethod Matrix
I am taking a statistics subject for psychology and the content has recently covered multitrait multimethod matrices (MTMM). I am trying to wrap my head around it. Just as an example, for a matrix ...
0
votes
1answer
74 views
Association between continuous input and categorical output
I have two independent continuous variables like Age, Price and an outcome variable like purchased or not purchased
Now, which test should I use to ascertain the association of continuous input ...
2
votes
1answer
198 views
A feature highly correlated with target variable
What if one of the predictor variables is highly correlated with the target variable (say 0.9), what should we do? Should we drop it or keep it to build the prediction model(classification or ...
1
vote
0answers
32 views
I have a data set of optimal values after simulations, How can I find if this dataset follows a specific pattern or any relation exists?
In the simulation I am conducting, I have a set of triangles and I select the optimal triangle based on my metric. After every simulation, I obtain an optimal triangle and I note down the lengths of ...
1
vote
0answers
18 views
correlated variables & model performance: optimal trade-off
on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift.
assuming little to medium ...
1
vote
0answers
155 views
Pearson correlation on two categorical variables
I am using the fourth-corner method in one of my papers (for those who need the name). The method was developed to test associations between variables in two datasets. In my case, the datasets ...
0
votes
0answers
3 views
Timeserieses Xi are noisy, and the noise depends on a timeseries Y. How do I remove that noise, to smooth the timeserieses Xi?
The noises in my timeserieses Xi are highly correlated on the timeseries Y. How would you approach the problem of removing that noise? Ideally I would like all the Xi curves to be smooth, as well as ...
0
votes
0answers
8 views
Is there a difference between correlation rules and sequential patterns?
In security I read correlation rules in SIEM systems while in data mining I read sequential patterns. They appear to be something similar, because in both cases the order of itemsets matter. Is there ...
1
vote
1answer
57 views
Visualization methods in R to examine correlation of labels against response
Question
What are some good plotting methods in R for examining the relationship between a target variable and various explanatory variables? In particular, I'm looking for visualization techniques ...
0
votes
1answer
23 views
Evaluating correlations between dichotomous/binary variables? What test?
I have 4 variables, each are dichotomous, 1 = in this group & 0 = not in this group (for each of the 4 groups).
I have read mixed accounts of whether ...
0
votes
1answer
37 views
What is the difference between causal discovery and inverse modeling?
I do not see these words used interchangeably, but they seem to be similar. In inverse modeling we are trying to find causal factors given an effect. In causal discovery, we are also looking for ...
2
votes
0answers
18 views
Which method to use to remove correlation between independent variables comprising of both categorical and numerical variables? [duplicate]
The independent variables in the dataset contains categorical variables such as
Gender ( 2 levels)
Mode of Shipment ( 3 levels)
Product Importance ( 4 levels)
and Numerical Variables such as
...
0
votes
3answers
126 views
Removing correlation between independent variables
If there are two continuous independent variables that show a high amount of correlation between them, can we remove this correlation by multiplying or dividing the values of one of the variables with ...
1
vote
0answers
22 views
Analysis of changing correlation
Given a correlation matrix M and after an event, M changes.
What is the best way to analyse the change? Verify significance (presumably T-test, ANOVA..)
Currently looking at the %, count & average ...
