Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
1
vote
1answer
13 views
Automatically Remove highly correlated features using Variance Inflation Factor?
I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated.
Multiple features can be ...
0
votes
0answers
15 views
Best technique for testing if an event causes a trend?
Example - A major holiday is coming up and the frequency of a product being purchased starts increasing significantly as it gets closer to the day of the holiday. Then after the holiday, the frequency ...
0
votes
1answer
23 views
how to deal with two high correlations feature which both has a low correlation with target
I am doing a prediction of house trade money.
Here is the correlation matrix of features whose correlations are larger than 0.3 as follows:
...
1
vote
1answer
75 views
What measures can I use to find correlation between categorical features and binary label?
For analyzing numerical features, we have correlation. What measures do we have to analyse the relevance of a categorical feature to the target value? If there isn't a direct measure, how can we ...
0
votes
3answers
34 views
How to find the relationship between two categorical variables?
I'm building a model and trying to get the relationship between two multi-level categorical variables.
For example, I want to know the relationship between race and likelihood of graduate from ...
0
votes
1answer
22 views
Using a feature in prediction even if it gets zero as p-value?
I created two binary classification based logistic regression models and I got these results:
Model 1:
Accuracy: 67.51%
AUC: 65.21%
Model 2:
Accuracy: 67,99%
AUC: 65,70%
The second ...
0
votes
1answer
22 views
What does coefficient mean for a binary independent variable in multiple linear regression?
Let's say I have a multiple linear regression model where my dependent variable, Y, is an integer. And, one of my independent variables --x1-- is binary --let's say either 0 or 1.
We know that sign ...
0
votes
0answers
9 views
create joint prob distribution or empirical relation for two variables
There are two variables, X1 and X2. The experimental study shows that they are highly correlated. Are there any reliable ways to create an empirical mapping(or equation) between X1 and X2. Assuming ...
7
votes
1answer
151 views
Generate predictions that are orthogonal (uncorrelated) to a given variable
I have an X matrix, a y variable, and another variable ORTHO_VAR. I need to predict the <...
1
vote
0answers
11 views
What kind of test should I use to determine correlation between the value of a commodity and consumer sentiment?
I’m currently studying data science and am trying to apply my skills to a small project.
Basically, I’ve collected data about a commodity’s value over two time periods (30 data points each in 4 week ...
1
vote
2answers
42 views
Establish relationship between two sets of data
I have two data sets - Product to Features and Products to Parts
...
1
vote
0answers
37 views
Pearson's Coefficient for classification problem
Can we use Pearson's Correlation Coefficient to calculate the correlation between features and target in the case of a classification problem?
0
votes
1answer
23 views
Correlation between Time Series Indicators ( Stock Prices )
I am new to time series analysis and I am currently tackling a stock market prediction problem.
I have a set of market indicators (such as Bollinger Bands, ADX etc) which are derived from the time ...
3
votes
2answers
88 views
Manual feature engineering based on the output
So, I'm working on a ML model that would have as potential predictors : age , a code for his city , his social status ( married / single and so on ) , number of his children and the output signed ...
3
votes
0answers
39 views
Fitting model to differenced time series
I have a time series on daily stock price of company(2013 data points).I took a first order difference and the following acf and pacf plots of the differenced series were obtained. However, I am ...
0
votes
1answer
74 views
Clustering by Distributions of Groups of observations
I have data as shown below. Like the groups shown A,B,C.. there are roughly 5400 groups of data and 15 observations for every group (I sampled 6-7 for each group and pasted them here).
...
0
votes
2answers
21 views
Causality analysis
I'm trying to find causes for my lack of sleep through data. I'm hoping to be able to input information throughout the day like "drank coffee", "Exercised", "meditation","what I ate for dinner"... and ...
1
vote
1answer
69 views
Dealing with correlated features when calculating permutation importance
I have implemented the permutation importance calculation as found here in my attempt to identify features that contribute little to my model's (Gradient Boosted Tree model) predictive power.
The ...
0
votes
1answer
113 views
How to measure correlation between several categorical features and a numerical label in Python?
I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product ...
1
vote
1answer
30 views
How to find correlation between time-series of different units?
I have 3 time-series data.
NDVI(normalized difference vegetation index) mean
Precipitation
Temperature
All of these have their own unit.
Now I want to find similarity/correlation between NDVI and ...
0
votes
1answer
88 views
How to select variables based on the mean correlation in a correlation matrix?
I have a set of independent variables and I am calculating the correlation matrix between them using the Pearson Correlation Coefficient in Python. A part of the matrix looks like this:
From this ...
0
votes
0answers
7 views
Identifying causation between content <> ad kpis
I have a bunch of content tagged with various types of content (e.g. political news, sports, opinion etc.) and a bunch of kpis related to ads placed next to this content (e.g click-throughs, purchases ...
1
vote
1answer
40 views
Machine learning algorithm that uses the Pearson or Spearman correlation?
I've come across linear and multiple regression, SVM, random forests. Does any know of a machine learning algorithm that uses the Pearson correlation or Spearman correlation?
Best,
Dave
0
votes
2answers
46 views
Trying to find the correlation between inputs and output
I have tried the pandas code for trying to find out the correlation between the output and the inputs I am feeding. Here is the code:
...
0
votes
2answers
699 views
Correlation between nominal categorical variables
I have two arrays, whose values are nominal categorical variables. Each element represents a zone of a city: in the first vector we have the class each zone belongs to (so these might also be seen as ...
0
votes
2answers
30 views
Hesitate to drop a feature
I used to tried making a fake correlation in my mocking dataset and found that if the score is more than 0.5 I can reduce feature to avoid singularity
In the given ...
0
votes
1answer
16 views
Determining the correlations between aggregated data and non aggregated data
I have a dataset which is essentially a list of lists produced by a sql query output. Here is what it looks like
...
0
votes
2answers
45 views
Building predictive model with low correlated data [closed]
I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
0
votes
1answer
34 views
Does this correlation make sense?
I'm trying to work on a ML project and I have a dataset and I'm trying to see if there is a correlation between some of the features in my dataset.
The dataset contains inspection notes for car parts ...
1
vote
3answers
63 views
How to find similarity of two series over time containing periodic trends?
Considering the data is received from a streaming source each second.How to distinguish if both the line graphs 'look' same/different in real time, statically, like the picture given below
Edit:
1....
2
votes
2answers
148 views
How to fill missing numeric if any value in a subset is missing, all other columns with the same subset are missing
There is a clear pattern that show for two separate subsets (set of columns); If one value is missing in a column, values of other columns in the same subset are missing for any row.
Here is a ...
1
vote
1answer
39 views
How to calculate temporary/periodic similarities of an increasing series in real time?
Considering there are two series over time and new data is added in the series over a gap of n second . The series might have periodic similarity/dis-similarity within themselves. How to calculate ...
1
vote
2answers
64 views
When creating a classification model, should predictors with little correlation to the response variable be included in the model?
I am building a predictive model designed to predict attrition within my organization. I am trying to decide whether to add certain predictors to my model. I used a Kruskal-Wallis rank sum test to ...
3
votes
1answer
98 views
In supervised learning, how to get info from correlation? [duplicate]
I am trying to build a classification model so I have tried to check the correlation between the features.
Here Loan_Status is my target variable.
I just don't know how to extract information from ...
1
vote
1answer
41 views
How is PACF analysis output related to LSTM ?
I was going through a recent paper “A Novel Hybrid Data-Driven Model for Daily Land Surface Temperature ForecastingUsing Long Short-Term Memory Neural Network Based on Ensemble Empirical ...
1
vote
0answers
9 views
Measuring Effects of Multiple Changes over Time
I'm writing a social media marketing plan for a website. I have several changes I am planning to implement (increasing posting rate, changing type of content posted, etc.) over an extended timeline to ...
1
vote
6answers
128 views
How to statistically prove that a column in a dataframe is not needed
I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
3
votes
2answers
54 views
Regression or Correlation for this RQ?
Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in ...
1
vote
0answers
103 views
Spearmanr on two pandas dataframes
I have two pandas dataframes that look about the same but with different information stored in them. My question will be about how to compare the two dataframes to ensure column and row match before ...
3
votes
1answer
53 views
Feature selection for time series prediction
I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables.
When calculating correlation between variables (e.g. Close price of Tesla vs ...
1
vote
0answers
40 views
Problems with Graphical Lasso
I'm trying to use the Graphical Lasso algorithm (more specifically the R package glasso) to find an estimated graph representing the connections between a set of nodes by estimating a precision matrix....
2
votes
2answers
31 views
Is there any logic to adding a threshold to see if two variables are related?
I have two variables $X$ and $Y$ given as tuples of $(x, y)$, and I want to see if there is a relationship between the two variables. I can do so by finding the correlation coefficient.
However, I ...
0
votes
1answer
22 views
automatic feature selection
I have a lot (thousands are possible) of automatically-generated ordinal features that i'd like to exploit , to differentiate between two classes. I'm looking for some measure that will select the ...
3
votes
1answer
91 views
Representing cyclical features as sin/cos components
I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc.
After some searching I decided to follow the advice here.
Now ...
0
votes
1answer
26 views
Time Posting Data Analysis
I work in a professional services company and would like to get some analytics on how discipline the fee earners post their times to the system may have an impact to the revenue.
One area that I am ...
1
vote
0answers
216 views
What is the relationship between correlation ratio and one-way Anova?
According to the answer to this post, it is recommended to use one-way anova to compute the dependence between a categorical and a numerical variable.
Besides, the second answer to this post says ...
1
vote
2answers
140 views
Random Forests with complementary features
In my dataset, I have 2 features that are not only correlated but that makes sense only in the presence of each other. For instance, one would be the number of times a task was attempted and the other ...
3
votes
1answer
199 views
Standard correlation coefficient of various datasets
I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure 1. Why is it so ?. Even the shapes differ for ...
0
votes
1answer
67 views
Chi-squared for continuous variables
I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into ...
3
votes
3answers
562 views
Is Pearson coefficient a good indicator of dependency between variables?
Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: "I would just calculate the Pearson correlation coefficient". That wasn't a ...
