Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
274
questions
0
votes
1answer
19 views
What type of correlation analysis would be appropriate in case of many variables (binary, numerical and text)?
I am approaching the field of machine learning for fake news detection. I built a dataset including features like information on punctuation, number of sentences, capital letters, urls, age of the ...
1
vote
1answer
21 views
How do I use the data I have to make predictions? [closed]
I've a set of CPUs (~100). I've the results of each CPU being benchmarked against a suite of benchmarks (~8). There is a specific task that interests me, and I need to find the optimal CPU for that ...
2
votes
1answer
33 views
Correlation among features (e.g. doc length, punctuation, … ) in classifying spam emails
I extracted some other features from my dataset regarding punctuation, capital letters, upper case words. I got these value:
looking at the correlation with my target variable (1=spam, 0=not spam), ...
1
vote
0answers
26 views
Low Pearson correlation coefficient between stable signals and high Pearson correlation coefficient between unstable signals
I calculated the Pearson correlation coefficient between two signals, that described the state of the unit. During normal operation of the unit, both signals were fairly stable and fluctuated very ...
0
votes
0answers
21 views
Correlations between target variables and other variables (binary)
I would like to calculate the correlation between the presence of uppercase, special punctuation, specific words, in texts labeled as fake/not fake.
In order to do it, I created some columns in my ...
0
votes
1answer
19 views
What statistical method should i use to find Correlation between number of days and AmountEarned
I am new to Data Science and I have a python data frame with Number of days, CountofJobs, and AmountEarned what statistical method should I use to find a correlation between Days and AmountEarned.
<...
1
vote
1answer
50 views
My corr() function in Python keeps resulting in an “ValueError: The truth value of a Series is ambiguous…” [closed]
I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, ...
1
vote
0answers
30 views
Handling highly correlated features [closed]
I have a data set of transactions and want to build a fraud detection model (classifier). Only 3 variables are given that could be used as input features. The number of transactions during past 3, 6 ...
1
vote
1answer
29 views
How to handle a valuable feature that is missing on 99\% of the samples in the data set?
Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this ...
2
votes
1answer
41 views
How to visualise a large correlation matrix?
I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21).
Now ...
0
votes
1answer
27 views
Determine relationship between users and age?
I would like to understand how to find an association between users, spam and email's age.
My dataset looks like as follows:
...
4
votes
2answers
75 views
Can i expect good results having low correlation attributes?
This was a question i saw in an interview for a data scientist position:
"Here is the following correlation heatmap that i got from my attributes. Regarding the correlation of each feature with ...
1
vote
2answers
39 views
is it better to correlate and encode or encode and correlate?
I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting ...
3
votes
2answers
71 views
How can I determine the relationship between spam and weekdays?
I am trying to check if there is a correlation between spam emails and weekdays.
My dataset looks like as follows:
...
0
votes
1answer
35 views
2
votes
1answer
77 views
Problem regarding calculating correlation approach?
There are 3 columns say product_id, product_type, and price_drop. ...
4
votes
3answers
211 views
Is autocorrelation of residuals a problem in machine learning?
Let's assume I have a random forest model and the residuals of the model are autocorrelated. Is this a problem?
As an example, let's assume I have two different random forest models, A and B, with a ...
1
vote
0answers
54 views
The Merits of Feature Reduction Routines
I am interested in learning what routine others use (if any) for Feature Reduction/Selection.
For example, If my data has several thousand features, I typically try {2,3,4} things right away depending ...
0
votes
0answers
21 views
Autocorrelation plot meaning!
I use pandas autocorrelation plot for a time series, and the result is:
How cany I inference from it?! This is my code :
...
0
votes
0answers
22 views
Is there a certain threshold over which to accept or reject predictors based on correlation values with the target variable?
I have been working on the Titanic dataset. After some feature manipulation, I printed out the correlation values between my target variable Survived and all the ...
0
votes
0answers
17 views
create and use weights in Python to perform weighted correlation and PCA
I need help with the principal componentsā analysis code below in 3 ways:
Write Code to create RIM (RAKE) weighting.
Iām trying to ensure that using the code below, I get exactly the same results than ...
0
votes
1answer
31 views
Finding the worst affected industry due to COVID in terms of unemployment
My goal is to find the worst affected industries from COVIDā19 in terms unemployment.
In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
1
vote
1answer
48 views
What conclusion can I get when the variable is influenced by other but there isn't any correlation?
I am doing an analytic exploratory analysis.
If the target is a continuous variable and the attributes are all categorical (discrete values), in order to know if exist any influence on the target from ...
4
votes
2answers
110 views
what is the difference in terms namely Correlation, correlated and collinearity?
A website says Correlation refers to an increase/decrease in a dependent variable with an increase/decrease in an independent variable. Collinearity refers to two or more independent variables acting ...
2
votes
2answers
64 views
Measure correlation for categorical vs continous variable
Given a variable which is categorical that depends on continuous variables, I would like to know how to check wether these continous variable explain the categorical one.
So:
...
0
votes
2answers
57 views
Can features negatively correlated with the target be used?
In feature selection (for a regression problem), can features that are negatively correlated with the target variable be chosen to predict the target?
I don't think negative correlation means the ...
1
vote
0answers
40 views
cor() function giving unexpected result for numeric data frame(Updated) [closed]
I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the ...
1
vote
0answers
80 views
Which statistical test you propose to assess whether there is significant effect of specified categorical variable (s) on a categorical outcome?
I have a table that I am trying to analyse.
...
1
vote
2answers
55 views
How to find correlation between categorical data and continuous data
I'm working on imputing null values in the Titanic dataset.
The 'Embarked' column has some. I do NOT want to just set them all to the most common value, ...
3
votes
0answers
60 views
How to find lagged cross correlation between time series?
I have 2 time series, $X$ and $Y$, and I'm trying to find the best lag range that correlates $X$ to $Y$ (find the amount(s) of lag of $X$ that best correlate to the target variable $Y$).
For instance, ...
4
votes
2answers
92 views
Negatively correlated features
Is it ok to use negatively correlated features in data modeling? Say I have features A and B that have a correlation coefficient ...
2
votes
1answer
55 views
Computing adjusted p-values in batches
Data
For simplicity I propose the following scheme:
I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous:
...
1
vote
0answers
24 views
Pairwise 3D object correlation between 2 objects
I have a dataset which contains 3D CT scans from different patients along with the segmenation masks of a certain organ. The 3D scans have been drawn each day for a period of 30 days for each patient. ...
2
votes
2answers
84 views
Checking linearity for a linear regression model?
I've read that there are various assumptions associated with a multiple linear regression model which you should check/validate before getting too excited about your model results.
One of these is the ...
0
votes
0answers
50 views
Anomaly Detection over multivariate data containing Nominal and numerical predictors
I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors.
Dataset has following pattern:
If we consider the below sample records, category_id, ...
1
vote
1answer
32 views
How do i perform feature selection for clustering?
I am new to Data science and trying to learn clustering? I have to partition the given dataset into different clusters into customer clusters based on their purchasing habits? How do I select the ...
0
votes
1answer
29 views
Zero-inflated independent feature in tree-based models
What is the best approach to include a zero-inflated continuous independent feature (e.g., 90% of the values are Zero, 10% are >0) in a Tree-based models (DT, random forest, gradient boosting. etc)....
13
votes
8answers
2k views
If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?
Let's say
A and B are correlated
A and C are correlated
B and C is uncorrelated
How is it possible for B and C to be uncorrelated when they are both correlated to A?
0
votes
1answer
57 views
Relationship between two continuous variables in time series data
I have a dataset that collects daily data based on transactions between two entities.
I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of ...
0
votes
0answers
16 views
Correlation between features in python
I have a dataset which has categorical variables as features. They are nominal in nature. One of the variable has 312 categories. I want to check how correlated the variables are, to check ...
0
votes
1answer
34 views
Chi-square test - how can I say if attributes are correlated?
I am experimenting a course's teorical contents on this dataset. After data cleaning, I am trying to use chi-square test. I wrote the following code:
...
0
votes
0answers
13 views
Correlation between numeric attributes
I am trying to do some correlation analysis on my dataset. After data cleansing, I have two columns of numeric values, Rating and CocoaPerc. Using Pearson's Correlation Coefficient, Sperman's Rank ...
1
vote
3answers
290 views
Feature importance in neural networks
Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, ...
0
votes
0answers
30 views
Exploratory statistics, how to idenify and remove driver (bias)
I am looking at customer data, and created frequency tables (+histograms) for customers with different professional statuses and what the best time is to reach them. Status ranges here from employed, ...
0
votes
0answers
27 views
Feature Selection with non-linear numerical and categorical variables
I have a dataset of 45 non-linear numerical values and 2 categorical values. I am making a feature selection to predict categorical variables one by one or together. I used the correlation ratio and ...
0
votes
0answers
21 views
What is meant by coefficient of determination? Further, is there any difference between r and muliple correlation coefficient?
In the article, Daniel J. Ozer says I demonstrate that D'Anļæ¾drade and Dart's conclusion does not require
rejecting the variance as a measure of variaļæ¾tion. I describe instead a second model for
...
1
vote
0answers
11 views
lags number in multivariate time series analysis correlation
I am trying to calculate the correlation coefficient(Pearson's r) of a financial time series $Y(t)$ and other exogenous variables $X_1(t),..., X_n(t)$. I am trying to understand the impact of my ...
1
vote
1answer
29 views
Question on ANOVA and Correlation/Association
I've been working on examining statistical relationships between variable:
Pearsons, Spearman's for continuous variables
Kendall's Tau, Cramer's V for ordinal/nominal variables.
I know there's many ...
0
votes
0answers
6 views
How strong would you rate the association between this nominal predictor variable and continuous response variable?
I've yet to find an easy-to-understand and easy-to-implement way to statistically model the strength of association between nominal data and continuous data. So I'm trying to get a rough idea visually ...
0
votes
1answer
21 views
Cross Correlation Between Input-Output Sine Waves
I am writing an algorithm to estimate the frequency transfer function of the system. For this, I want to use the Cross-correlation Between Input-Output Sine Waves method. There are a few things I don'...
