Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
0
votes
1answer
19 views

What type of correlation analysis would be appropriate in case of many variables (binary, numerical and text)?

I am approaching the field of machine learning for fake news detection. I built a dataset including features like information on punctuation, number of sentences, capital letters, urls, age of the ...
1
vote
1answer
21 views

How do I use the data I have to make predictions? [closed]

I've a set of CPUs (~100). I've the results of each CPU being benchmarked against a suite of benchmarks (~8). There is a specific task that interests me, and I need to find the optimal CPU for that ...
2
votes
1answer
33 views

Correlation among features (e.g. doc length, punctuation, … ) in classifying spam emails

I extracted some other features from my dataset regarding punctuation, capital letters, upper case words. I got these value: looking at the correlation with my target variable (1=spam, 0=not spam), ...
1
vote
0answers
26 views

Low Pearson correlation coefficient between stable signals and high Pearson correlation coefficient between unstable signals

I calculated the Pearson correlation coefficient between two signals, that described the state of the unit. During normal operation of the unit, both signals were fairly stable and fluctuated very ...
0
votes
0answers
21 views

Correlations between target variables and other variables (binary)

I would like to calculate the correlation between the presence of uppercase, special punctuation, specific words, in texts labeled as fake/not fake. In order to do it, I created some columns in my ...
0
votes
1answer
19 views

What statistical method should i use to find Correlation between number of days and AmountEarned

I am new to Data Science and I have a python data frame with Number of days, CountofJobs, and AmountEarned what statistical method should I use to find a correlation between Days and AmountEarned. <...
1
vote
1answer
50 views

My corr() function in Python keeps resulting in an “ValueError: The truth value of a Series is ambiguous…” [closed]

I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, ...
1
vote
0answers
30 views

Handling highly correlated features [closed]

I have a data set of transactions and want to build a fraud detection model (classifier). Only 3 variables are given that could be used as input features. The number of transactions during past 3, 6 ...
1
vote
1answer
29 views

How to handle a valuable feature that is missing on 99\% of the samples in the data set?

Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this ...
2
votes
1answer
41 views

How to visualise a large correlation matrix?

I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21). Now ...
0
votes
1answer
27 views

Determine relationship between users and age?

I would like to understand how to find an association between users, spam and email's age. My dataset looks like as follows: ...
4
votes
2answers
75 views

Can i expect good results having low correlation attributes?

This was a question i saw in an interview for a data scientist position: "Here is the following correlation heatmap that i got from my attributes. Regarding the correlation of each feature with ...
1
vote
2answers
39 views

is it better to correlate and encode or encode and correlate?

I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting ...
3
votes
2answers
71 views

How can I determine the relationship between spam and weekdays?

I am trying to check if there is a correlation between spam emails and weekdays. My dataset looks like as follows: ...
0
votes
1answer
35 views

Estimate correlation in Python

I have a dataset with labels and usernames: ...
2
votes
1answer
77 views

Problem regarding calculating correlation approach?

There are 3 columns say product_id, product_type, and price_drop. ...
4
votes
3answers
211 views

Is autocorrelation of residuals a problem in machine learning?

Let's assume I have a random forest model and the residuals of the model are autocorrelated. Is this a problem? As an example, let's assume I have two different random forest models, A and B, with a ...
1
vote
0answers
54 views

The Merits of Feature Reduction Routines

I am interested in learning what routine others use (if any) for Feature Reduction/Selection. For example, If my data has several thousand features, I typically try {2,3,4} things right away depending ...
0
votes
0answers
21 views

Autocorrelation plot meaning!

I use pandas autocorrelation plot for a time series, and the result is: How cany I inference from it?! This is my code : ...
0
votes
0answers
22 views

Is there a certain threshold over which to accept or reject predictors based on correlation values with the target variable?

I have been working on the Titanic dataset. After some feature manipulation, I printed out the correlation values between my target variable Survived and all the ...
0
votes
0answers
17 views

create and use weights in Python to perform weighted correlation and PCA

I need help with the principal components’ analysis code below in 3 ways: Write Code to create RIM (RAKE) weighting. I’m trying to ensure that using the code below, I get exactly the same results than ...
0
votes
1answer
31 views

Finding the worst affected industry due to COVID in terms of unemployment

My goal is to find the worst affected industries from COVID—19 in terms unemployment. In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
1
vote
1answer
48 views

What conclusion can I get when the variable is influenced by other but there isn't any correlation?

I am doing an analytic exploratory analysis. If the target is a continuous variable and the attributes are all categorical (discrete values), in order to know if exist any influence on the target from ...
4
votes
2answers
110 views

what is the difference in terms namely Correlation, correlated and collinearity?

A website says Correlation refers to an increase/decrease in a dependent variable with an increase/decrease in an independent variable. Collinearity refers to two or more independent variables acting ...
2
votes
2answers
64 views

Measure correlation for categorical vs continous variable

Given a variable which is categorical that depends on continuous variables, I would like to know how to check wether these continous variable explain the categorical one. So: ...
0
votes
2answers
57 views

Can features negatively correlated with the target be used?

In feature selection (for a regression problem), can features that are negatively correlated with the target variable be chosen to predict the target? I don't think negative correlation means the ...
1
vote
0answers
40 views

cor() function giving unexpected result for numeric data frame(Updated) [closed]

I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the ...
1
vote
0answers
80 views
1
vote
2answers
55 views

How to find correlation between categorical data and continuous data

I'm working on imputing null values in the Titanic dataset. The 'Embarked' column has some. I do NOT want to just set them all to the most common value, ...
3
votes
0answers
60 views

How to find lagged cross correlation between time series?

I have 2 time series, $X$ and $Y$, and I'm trying to find the best lag range that correlates $X$ to $Y$ (find the amount(s) of lag of $X$ that best correlate to the target variable $Y$). For instance, ...
4
votes
2answers
92 views

Negatively correlated features

Is it ok to use negatively correlated features in data modeling? Say I have features A and B that have a correlation coefficient ...
2
votes
1answer
55 views

Computing adjusted p-values in batches

Data For simplicity I propose the following scheme: I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous: ...
1
vote
0answers
24 views

Pairwise 3D object correlation between 2 objects

I have a dataset which contains 3D CT scans from different patients along with the segmenation masks of a certain organ. The 3D scans have been drawn each day for a period of 30 days for each patient. ...
2
votes
2answers
84 views

Checking linearity for a linear regression model?

I've read that there are various assumptions associated with a multiple linear regression model which you should check/validate before getting too excited about your model results. One of these is the ...
0
votes
0answers
50 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
1
vote
1answer
32 views

How do i perform feature selection for clustering?

I am new to Data science and trying to learn clustering? I have to partition the given dataset into different clusters into customer clusters based on their purchasing habits? How do I select the ...
0
votes
1answer
29 views

Zero-inflated independent feature in tree-based models

What is the best approach to include a zero-inflated continuous independent feature (e.g., 90% of the values are Zero, 10% are >0) in a Tree-based models (DT, random forest, gradient boosting. etc)....
13
votes
8answers
2k views

If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

Let's say A and B are correlated A and C are correlated B and C is uncorrelated How is it possible for B and C to be uncorrelated when they are both correlated to A?
0
votes
1answer
57 views

Relationship between two continuous variables in time series data

I have a dataset that collects daily data based on transactions between two entities. I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of ...
0
votes
0answers
16 views

Correlation between features in python

I have a dataset which has categorical variables as features. They are nominal in nature. One of the variable has 312 categories. I want to check how correlated the variables are, to check ...
0
votes
1answer
34 views

Chi-square test - how can I say if attributes are correlated?

I am experimenting a course's teorical contents on this dataset. After data cleaning, I am trying to use chi-square test. I wrote the following code: ...
0
votes
0answers
13 views

Correlation between numeric attributes

I am trying to do some correlation analysis on my dataset. After data cleansing, I have two columns of numeric values, Rating and CocoaPerc. Using Pearson's Correlation Coefficient, Sperman's Rank ...
1
vote
3answers
290 views

Feature importance in neural networks

Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, ...
0
votes
0answers
30 views

Exploratory statistics, how to idenify and remove driver (bias)

I am looking at customer data, and created frequency tables (+histograms) for customers with different professional statuses and what the best time is to reach them. Status ranges here from employed, ...
0
votes
0answers
27 views

Feature Selection with non-linear numerical and categorical variables

I have a dataset of 45 non-linear numerical values and 2 categorical values. I am making a feature selection to predict categorical variables one by one or together. I used the correlation ratio and ...
0
votes
0answers
21 views

What is meant by coefficient of determination? Further, is there any difference between r and muliple correlation coefficient?

In the article, Daniel J. Ozer says I demonstrate that D'An￾drade and Dart's conclusion does not require rejecting the variance as a measure of varia￾tion. I describe instead a second model for ...
1
vote
0answers
11 views

lags number in multivariate time series analysis correlation

I am trying to calculate the correlation coefficient(Pearson's r) of a financial time series $Y(t)$ and other exogenous variables $X_1(t),..., X_n(t)$. I am trying to understand the impact of my ...
1
vote
1answer
29 views

Question on ANOVA and Correlation/Association

I've been working on examining statistical relationships between variable: Pearsons, Spearman's for continuous variables Kendall's Tau, Cramer's V for ordinal/nominal variables. I know there's many ...
0
votes
0answers
6 views

How strong would you rate the association between this nominal predictor variable and continuous response variable?

I've yet to find an easy-to-understand and easy-to-implement way to statistically model the strength of association between nominal data and continuous data. So I'm trying to get a rough idea visually ...
0
votes
1answer
21 views

Cross Correlation Between Input-Output Sine Waves

I am writing an algorithm to estimate the frequency transfer function of the system. For this, I want to use the Cross-correlation Between Input-Output Sine Waves method. There are a few things I don'...

1
2 3 4 5 6