Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
1
vote
0answers
4 views
Measuring Effects of Multiple Changes over Time
I'm writing a social media marketing plan for a website. I have several changes I am planning to implement (increasing posting rate, changing type of content posted, etc.) over an extended timeline to ...
0
votes
3answers
44 views
How to statistically prove that a column in a dataframe is not needed
I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
3
votes
2answers
13 views
Regression or Correlation for this RQ?
Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in ...
0
votes
0answers
16 views
Visualizing correlation with large number of features
Often, visualization is touted as a prerequisite to model building, with the purpose of identifying features which are correlated with each other (think NxN correlation grid graph). The correlated ...
1
vote
0answers
13 views
Spearmanr on two pandas dataframes
I have two pandas dataframes that look about the same but with different information stored in them. My question will be about how to compare the two dataframes to ensure column and row match before ...
0
votes
0answers
8 views
How to find correlation among multiple attributes in group by dataframe object?
I have a data frame with following attributes :
CP - Counting point of vehicles
A-Junction - Starting node of a road
...
2
votes
1answer
25 views
Feature selection for time series prediction
I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables.
When calculating correlation between variables (e.g. Close price of Tesla vs ...
1
vote
0answers
15 views
Problems with Graphical Lasso
I'm trying to use the Graphical Lasso algorithm (more specifically the R package glasso) to find an estimated graph representing the connections between a set of nodes by estimating a precision matrix....
0
votes
0answers
9 views
Robust method to extract “pure” part of a variable
I am trying to perform a multi linear regression model:
$$y_i = β_0 + β_1x_{i1} + β_2x_{i2} +... + β_px_{ip} + ε_i$$
where $$x_{i1}, x_{i2}, ..., x_{ip}$$ are highly correlated with each other (VIFs ...
1
vote
2answers
28 views
Is there any logic to adding a threshold to see if two variables are related?
I have two variables $X$ and $Y$ given as tuples of $(x, y)$, and I want to see if there is a relationship between the two variables. I can do so by finding the correlation coefficient.
However, I ...
0
votes
0answers
10 views
Looking for correlation between a big set of variables: divided in groups or all together?
I have a tiny classified dataset (2 classes and over 20 samples for each class) where I would like to find the highest pair-correlated variables. I doubt if I should mix both groups of samples and run ...
0
votes
1answer
16 views
automatic feature selection
I have a lot (thousands are possible) of automatically-generated ordinal features that i'd like to exploit , to differentiate between two classes. I'm looking for some measure that will select the ...
3
votes
1answer
54 views
Representing cyclical features as sin/cos components
I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc.
After some searching I decided to follow the advice here.
Now ...
0
votes
1answer
21 views
Time Posting Data Analysis
I work in a professional services company and would like to get some analytics on how discipline the fee earners post their times to the system may have an impact to the revenue.
One area that I am ...
1
vote
0answers
46 views
What is the relationship between correlation ratio and one-way Anova?
According to the answer to this post, it is recommended to use one-way anova to compute the dependence between a categorical and a numerical variable.
Besides, the second answer to this post says ...
0
votes
0answers
362 views
Finding the correlation between categorical variables
I'm working with a dataset of roughly 40000 files, each of which is the full text of a novel. I've found the percentages of words in each novel that belong to different mutually exclusive categories ...
0
votes
0answers
10 views
Correlation analysis and augmented dickey fuller test
I am performing the correlation analysis for each of the sectors (consumer discretionary, consumer staples, financials, technology, etc.) and the MSCI ACWI index. I am using their daily return data to ...
0
votes
0answers
31 views
Regression algorithm for weak correlated target and features
I have a regression problem in hand. Dataset have 20 predictors and 1 target. Target is continuous and predictors are both categorical and continous. I performed a correlation test between continous ...
0
votes
0answers
35 views
How to Classify X Column to the correlated Y Column?
I'm having trouble to attack this "classification" problem, with some pattern recognition.
I need to classify a whole X column to the correlated Y column.
For example,
Classify the data in column X2 ...
1
vote
2answers
73 views
Random Forests with complementary features
In my dataset, I have 2 features that are not only correlated but that makes sense only in the presence of each other. For instance, one would be the number of times a task was attempted and the other ...
0
votes
1answer
61 views
Standard correlation coefficient of various datasets
I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure 1. Why is it so ?. Even the shapes differ for ...
0
votes
1answer
27 views
Chi-squared for continuous variables
I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into ...
3
votes
3answers
425 views
Is Pearson coefficient a good indicator of dependency between variables?
Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: "I would just calculate the Pearson correlation coefficient". That wasn't a ...
0
votes
1answer
117 views
How do i interpret this correlation
Does this mean that as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college
Any additional things i can interpret from below ?...
2
votes
1answer
33 views
Discarding correlation among inputs in a neural network
I am working on a problem with 4 inputs and 1 continuous output variable. The sum of all values of the 4 input variables is always 1.
a1+a2+a3+a4=1
So, they are correlated.
My question is: ...
1
vote
0answers
30 views
Adjusted coefficient pearson as CNN loss function
I am trying to create a CNN (using python Keras) that runs on medical images of cancer and classifies them as cancerous or not. (Sample sizes vary between 30,000-280,000 on each trial run).
As an ...
1
vote
1answer
19 views
How do different models take in account independence/dependence of features?
How do different models take in account independence/dependence of features?
I've only found very rough "pointers" as to what assumptions different models make about the independence/dependence of ...
1
vote
0answers
23 views
Is it the job of the modelling approach to consider or not consider independence/dependence of features? Or should it be known before-hand?
I'm confused,
Is it the job of the modelling approach to consider or not consider independence/dependence of features? Or should it be known before-hand?
If I know nothing about independence/...
2
votes
2answers
135 views
Why is a correlation matrix symmetric?
I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix.
...
0
votes
1answer
43 views
What are some data sources with strong positive and negative correlations in feature and sample space?
For example, financial data has in some cases strong positive and negative correlations between different stocks (feature space) and also strong positive and negative correlations between specific ...
0
votes
0answers
16 views
Making sense of kernel canonical correlation analysis weights in the original space?
Kernel Canonical Correlation Analysis offers a non-linear alternative solution to the classical Canonical Correlation Analysis by first projecting the data into a higher-dimensional feature space. A ...
0
votes
0answers
63 views
Correlated features and double dipping
I am performing a machine learning analysis of MRI data. I have an extremely large number of features, thus I decided to remove highly correlated features and I decided to perform this step ...
0
votes
0answers
17 views
Algorithms affected by multicollinearity due to correlation between independent features
Please help me in understanding which and all machine learning get affected by Multi-collinearity due to the correlation between the independent features.
0
votes
2answers
67 views
How do I determine if variables are correlated? Is it simply a mathematical calculation?
I'm self learning data science so bear with me as I try and make my question as clear as possible. Lets assume I have a dataset of a dependent variable y and some independent variables x1, x2 and x3 ...
2
votes
1answer
46 views
Generalization of Correlation Coefficient
The correlation coefficient tells me how two variables (sequences of numbers) are correlated with each other. Does it generalize to non-linear scenarios? How could one more generally measure the ...
-1
votes
1answer
73 views
Clustering a contingency table
I have a contingency table listing individuals with certain traits. For the sake of simplicity, let's say the table has individuals in rows, and the type of food they like in columns:
...
0
votes
0answers
25 views
Correlation between products
I have a huge set of past sales data and I would like to know if there are correlations between the products (for example if a client buys product A he is 80% more likely to buy product B... If a ...
0
votes
0answers
17 views
Fill missing continuous values using Machine Learning models
Is it a good idea to fill missing data with ML models, like Linear Regression or KNN, that use other highly correlated features to make predictions?
1
vote
0answers
82 views
Is it advisable or okay to view heatmap of categorical features?
As the title suggested. By default sns.heatmap will not include categorical features which are not factorized.
Since factorized categorical features do not behave like a scale (eg. Quality: 0-10), I ...
0
votes
0answers
16 views
Weight visualization for the Deep Canonical Correlation Analysis
In the case of the linear Canonical Correlation Analysis (CCA), one can easily visualize the weight vectors. This helps with intuition about which parts of the input are mainly responsible for the ...
0
votes
0answers
347 views
Finding correlation between categorical feature and the continuous features
First, I have some missing values in the dataset. To find out the correlation among variables I need to fill the data first which I did. Next I have to find correlation between categorical feature and ...
1
vote
0answers
98 views
Autocorrelation of a sine wave
I would like to know the meaning of an autocorrelation graph of a sine wave. When the time lag is 0, then the autocorrelation should give the highest value of 1 since a copy of the signal is ...
0
votes
0answers
4 views
Quantiles for queuing processes with exponential arrivals and Pareto departures
I am generating the data from process
$X_{n}= X_{n-1} + U_n - W_n $
where
$ U_n= [U_{n1}, U_{n2}]$
is bivariate exponential process with correlation
and $W_n= [W_{n1}, W_{n2}]$
is bivariate Pareto ...
0
votes
0answers
35 views
How should I unify this type of variable?
I have the following problem: I have a variable that is composed by N levels and I want to find out if other numeric and continuous variable is correlated with this ...
0
votes
1answer
316 views
Is it good practice to always remove highly correlated variables?
1- Would it always be beneficial to remove highly correlated features prior to training a model? If not, why not.
2- Would you perform One Hot encoding where applicable, prior to removing highly ...
3
votes
1answer
629 views
How does multicollinearity affect neural networks?
Multicollinearity is a problem for linear regression because the results become unstable / depend too much on single elements (source).
(Also, the inverse of $X^TX$ doesn't exist so the standard OLS ...
2
votes
3answers
472 views
1
vote
2answers
51 views
Calculating correlation of slightly out of sync data
I am trying to do some analysis on some data that comes from special glasses that track a few things including pupil size and gaze velocity. I would like to calculate the correlation between two ...
4
votes
3answers
147 views
which neural network topology to learn correlations between time series?
I have two (or more in principle) 1xN time series, and I would like to train a NN to predict the next value of both. I can arrange them as a 2xN matrix and feed a window from this matrix as input to ...
0
votes
0answers
152 views
Correlation of words and score
Say I have two fields:
"Comments" (type text) and "Score" (type float, from 0 to 5)
Sample data like:
"I appreciate this ...
