Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
307
questions
0
votes
0answers
6 views
Which method to use to remove correlation between independent variables comprising of both categorical and numerical variables?
The independent variables in the dataset contains categorical variables such as
Gender ( 2 levels)
Mode of Shipment ( 3 levels)
Product Importance ( 4 levels)
and Numerical Variables such as
...
1
vote
0answers
16 views
Analysis of changing correlation
Given a correlation matrix M and after an event, M changes.
What is the best way to analyse the change? Verify significance (presumably T-test, ANOVA..)
Currently looking at the %, count & average ...
0
votes
0answers
8 views
Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?
I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
0
votes
0answers
12 views
At what correlation factor we consider two features highly correlated so that it is safe to drop one feature for supervised learning?
At what correlation factor we consider two features highly correlated so that it is safe to drop one feature for supervised learning?
It is said that in supervised learning we should remove the high ...
1
vote
0answers
22 views
How to get correlation between the categories of two categorical variable?
I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
0
votes
0answers
11 views
Relative scoring of entities
I'm storing a set of house data in my Elasticsearch database with various attributes for each entity (such as price, number of bathrooms, sqft, etc..) I want to create a basic ranking engine that ...
0
votes
1answer
15 views
How to compute time-lagged correlation between two variables with many examples at each time t?
I have a dictionary of following form:
datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year
So, essentially, I'm trying to get an aggregate ...
0
votes
0answers
25 views
Find top features that determine movie rank
I am trying to analyze a movie dataset in order to find the specific features which determine whether or not the movie is in the top-10 movies of the year (or likewise the worst-10 movies of the year)....
1
vote
0answers
28 views
Should I drop Nan before before creating a scatter plot and computing the correlation matrix?
I have some NaN values in my data and I cannot replace it with median or something else.
Now I have to check out the correlation, using scatter plot and Pearson correlation.
Should I drop these NaNs ...
0
votes
1answer
13 views
Are there readily available models that can handle conditional correlation?
I've been working my way through the features of the Kaggle House Prices dataset (Note: this is a non-ranking entry, so this is just for exercises), and I've found a couple situations where there is a ...
0
votes
0answers
13 views
What are practical benefits of the various distance metrics in scipy?
I’m looking for a information about distance metrics.
Python’s scipy gives many different metrics, and I’d like to know more about their practical use.
In particular, I’m trying to understand the ...
1
vote
1answer
34 views
Heat map and correlation among variables [closed]
I would have a question on heat map and correlation among variables.
I created this heat map, looking at possible correlation among variables and target. I got very small values.
I wanted to set a ...
1
vote
1answer
24 views
Dimensionality reduction to correlate large number of variables
So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a ...
1
vote
0answers
37 views
What can I infer from a linear correlation of regression coefficients?
I am working on a dataset for classification where each observation is a series of values of a certain measurement $Y$ for a fixed range of values of measurement $X$ (i.e. a discrete mapping from $A \...
1
vote
0answers
17 views
Are convolutions in deep learning associative?
Let's denote "convolution in deep learing" as "convolution-deep", and "convolution in math or signal processing" as "convolution-math".
As we all know, ...
3
votes
1answer
28 views
Does a Convolutional Layer in a Neural Network learn the correlation between its input signals via its kernel?
I am interested in the theory behing what a convolutional neural network learns with its convolutional operations. I think it learns (useful) kernels which measure the correlation between its input ...
0
votes
1answer
21 views
4
votes
1answer
86 views
How to make sure that the features learned by a neural network are not correlated?
Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the ...
1
vote
1answer
26 views
0
votes
1answer
24 views
Chi-Squared test: ok for selecting significant features?
I would have a question on the contingency table and its results.
I was performing this analysis on names starting with symbols as a possible feature, getting the following values:
...
3
votes
1answer
187 views
Features selection in imbalanced dataset
I have some doubts regarding an analysis. I have a dataset with class imbalance. I am trying to investigate some information from that data, e.g., how many urls contain http or https protocols.
My ...
0
votes
0answers
19 views
Correlation Study to Determine Weights Of Fields
I have several input fields, and the content for each field can either be correct or incorrect. These fields are then sent to a black-boxed function (which I can’t control), and the output of the ...
0
votes
0answers
8 views
Correlation coefficient for weighted value pairs
I want to obtain an appropriate correlation statistic for sparse matrices, in two different cases.
Case 1
A small rectangular matrix (size: 3-20 elements long or wide) with a single value per row, per ...
0
votes
0answers
24 views
Machine Learning and stability of correlations
I need some help improving my understanding. As far as I know many (if not all) ML algorithms assume time stationarity, that means that the sampled distribution should not change over time.
On a ...
0
votes
0answers
10 views
How to decompose values in a table
I have a table of values like this one:
Age
Population size
# of individuals who like apples
# of individuals who like oranges
Children
100
88
73
Adult
50
32
25
I am looking for a way to decompose ...
0
votes
1answer
25 views
Correlation/distance between sparse vectors
I am looking for a metric for comparing gene count tables. These are long columns of data (a few millions genes by a few dozen samples), with all non-negative entries, about 90% of which are zeros. ...
1
vote
1answer
28 views
Remove correlated features before or after splitting test and training set?
I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it ...
0
votes
1answer
27 views
Finding cause and effect correlation in a dataset
What methods might one use in order to find and analyze cause-and-effect in a dataset? An example might be something like plotting this:
...
1
vote
0answers
20 views
High correlation between the independent and dependent variables but low performance of regression model
I have a dataset of 4900 rows and 2060 feature. I calculated the correlation using kendall method between the dependent and independent features, and found out that 5 of these features are having a ...
2
votes
0answers
23 views
Kendall rank correlation coefficient's p-value
I'm trying to compute a p-value for a two tailed test following Wikipedia formula which indicates that:
one computes Z, and finds the cumulative probability for a standard
normal distribution at -|Z|....
1
vote
1answer
63 views
PCA, covariance, eigenvector matrix and rotation [closed]
I am following the Coursera NLP specialization, and in particular the lab "Another explanation about PCA" in Course 1 Week 3.
From the lab, I recovered the following code. It creates 2 ...
0
votes
0answers
6 views
Autocorrelation of two functions multiplied and raised to arbitrary powers
Given a signal $A$ and a signal $B$ with autocorrelation times of $\tau_A$ and $\tau_B$, respectively, where $\tau_A > \tau_B$, is there any general statement that can be made about the ...
3
votes
1answer
118 views
Pearson correlation with data sets that have values on different scales [closed]
I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have ...
1
vote
1answer
29 views
what is the best to do with highly correlated features?
In my data set 2 features C1 and C2 are highly correlated. I did the following steps. Could you please let me know if it is ...
0
votes
0answers
10 views
In a dataset, how to detect simple relationship between two columns (datatype not restricted to numerical values)?
Suppose, I have a dataset with the following features:
(FirstName, LastName, FullName, SchoolJoiningDate, SchoolGraduationDate)
My aim is to validate the data before feeding it into the ML pipeline. ...
2
votes
1answer
17 views
What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?
I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but ...
0
votes
0answers
17 views
Multi-Collinearity in Classification Problems
I have a "Small data matrix" of scraped data from multiple websites trying to account for sentiment towards certain cellphone models. The matrix includes independent variables such as "...
0
votes
2answers
28 views
Can we consider high correlation to be a good predictor?
The problem of predicting the daily number of COVID-19 cases is indeed challenging and many (external) factors should be taken into account to come up with a reasonable predictor. However, we have ...
1
vote
1answer
22 views
How do I use the data I have to make predictions? [closed]
I've a set of CPUs (~100). I've the results of each CPU being benchmarked against a suite of benchmarks (~8). There is a specific task that interests me, and I need to find the optimal CPU for that ...
2
votes
1answer
38 views
Correlation among features (e.g. doc length, punctuation, … ) in classifying spam emails
I extracted some other features from my dataset regarding punctuation, capital letters, upper case words. I got these value:
looking at the correlation with my target variable (1=spam, 0=not spam), ...
2
votes
1answer
45 views
How to interpret Low Pearson correlation coefficient between stable signals and high Pearson correlation coefficient between unstable signals?
I calculated the Pearson correlation coefficient between two signals, that described the state of the unit. During normal operation of the unit, both signals were fairly stable and fluctuated very ...
0
votes
1answer
31 views
What statistical method should i use to find Correlation between number of days and AmountEarned
I am new to Data Science and I have a python data frame with Number of days, CountofJobs, and AmountEarned what statistical method should I use to find a correlation between Days and AmountEarned.
<...
1
vote
1answer
454 views
My corr() function in Python keeps resulting in an “ValueError: The truth value of a Series is ambiguous…” [closed]
I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, ...
1
vote
0answers
35 views
Handling highly correlated features [closed]
I have a data set of transactions and want to build a fraud detection model (classifier). Only 3 variables are given that could be used as input features. The number of transactions during past 3, 6 ...
1
vote
1answer
31 views
How to handle a valuable feature that is missing on 99\% of the samples in the data set?
Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this ...
2
votes
1answer
177 views
How to visualise a large correlation matrix?
I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21).
Now ...
0
votes
1answer
30 views
Determine relationship between users and age?
I would like to understand how to find an association between users, spam and email's age.
My dataset looks like as follows:
...
4
votes
2answers
86 views
Can i expect good results having low correlation attributes?
This was a question i saw in an interview for a data scientist position:
"Here is the following correlation heatmap that i got from my attributes. Regarding the correlation of each feature with ...
1
vote
2answers
158 views
is it better to correlate and encode or encode and correlate?
I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting ...
3
votes
2answers
76 views
How can I determine the relationship between spam and weekdays?
I am trying to check if there is a correlation between spam emails and weekdays.
My dataset looks like as follows:
...
