Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
0
votes
1answer
131 views

Feature Engineering - correlation with binary outcome - Titanic Dataset - Ticket feature

I am currently building my first machine learning model using the titanic dataset. After the data exploration, I decided to focus my attention on the 'Ticket' feature. One thing I have noticed about ...
0
votes
0answers
21 views

If a feature is highly correlated to the target variable, would it also rank high when performing feature importance using a Random Forest Classifier?

I am trying to understand the relationship between pairwise correlation of variables in a dataset AND the feature importance of these variables using a Random Forest Classifier.
4
votes
2answers
452 views

How to identify recurring patterns in this set of transactional data

I'm working on a dataset of banking transactions and would like to find recurrent transactions. I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa....
0
votes
1answer
38 views

Time Posting Data Analysis

I work in a professional services company and would like to get some analytics on how discipline the fee earners post their times to the system may have an impact to the revenue. One area that I am ...
2
votes
1answer
208 views

Feature selection for time series prediction

I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables. When calculating correlation between variables (e.g. Close price of Tesla vs ...
1
vote
1answer
42 views

Find feature categories associated with a specific target category

I have a Dataset with three columns. Products (up to 200). The quality checks that have not been conducted at the final quality check. (Up to 70 different Quality Control Measures) The result of the ...
0
votes
0answers
14 views

Best method for calculating all correlations in a dataset?

First of all, to be clear, I don't want all correlations in the dataset, I just want to find all high correlations between the response variable and the features. I posted this question to reddit on '...
1
vote
1answer
27 views

Why is pearson correlation popular if it detects only linear correlation?

Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
1
vote
0answers
14 views

Feature creation: Problem with correlated features?

I recently started to read about feature creation. I've seen some general guidelines although I am not really sure if they are completely true, for example: 1 - Linear classifiers for binary ...
2
votes
1answer
33 views

Which stage should the correlation analysis be done?

I was thinking about it, but I couldn't find a logical explanation. Mostly im following below steps after data become ready: Correlation analysis and elimination Apply dummy if categorical variables ...
0
votes
1answer
54 views

What statistical method should i use to find Correlation between number of days and AmountEarned

I am new to Data Science and I have a python data frame with Number of days, CountofJobs, and AmountEarned what statistical method should I use to find a correlation between Days and AmountEarned. <...
1
vote
1answer
145 views

Relationship between two continuous variables in time series data

I have a dataset that collects daily data based on transactions between two entities. I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of ...
1
vote
1answer
47 views

Should I drop Nan before before creating a scatter plot and computing the correlation matrix?

I have some NaN values in my data and I cannot replace it with median or something else. Now I have to check out the correlation, using scatter plot and Pearson correlation. Should I drop these NaNs ...
3
votes
2answers
169 views

Similarity Measure of Simulated Time Series vs Observed time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
0
votes
3answers
61 views

How to find correlations between data over time?

I have daily data of my sales and a daily update of how the COVID-19 cases increase. My daily sales contains information on my customer and on my product. My end goal would be to see if certain ...
6
votes
3answers
853 views

Correlation vs Multicollinearity

I have been taught to check correlation matrix before going for any algorithm. I have a few questions around the same: Pearson Correlation is for numerical variables only. What if we have to check ...
2
votes
2answers
161 views

Regression or Correlation for this RQ?

Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in ...
0
votes
0answers
51 views

Predict multiple labels when labels are of mixed type: floating point and categorical

I was provided with a data file which looks like this: ...
1
vote
0answers
19 views

Is there a "tree-based-correlation" for tree-based algorithms?

Although correlated features are not a big issue when training tree-based models, they spoil model explainability. When several features correlate, sometimes they may be picked at random. Then their ...
1
vote
1answer
33 views

Get an equation from the relationship among variables

I have an agricultural dataset of crop yield (in kilograms) under environmental conditions such as humidity, rainfall, sunlight over few years. I want to find out relationship between the ...
1
vote
1answer
18 views

How to test whether data is clustered wrt. subcategories?

I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
0
votes
0answers
11 views

Event correlation in processes

I have a dataset with roughly 100 events per product (150 products). So each product has these 100 events. The last event is a quality check which is either red or green and which is either due or on ...
3
votes
1answer
605 views

SelectKBest and Correlation returns me excatly same feature selection. How?

Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But ...
1
vote
1answer
386 views

Is it advisable or okay to view heatmap of categorical features?

As the title suggested. By default sns.heatmap will not include categorical features which are not factorized. Since factorized categorical features do not behave like a scale (eg. Quality: 0-10), I ...
0
votes
2answers
39 views

If I have two variables with strong correlation, should I delete one and leave the other in my data

I have a large dataset, where I should make a binary prediction. The fact is that, after analyzing the data, I found that some variables are positively correlated to each other. So, I was wondering ...
1
vote
1answer
1k views

Correlation Matrix Issues: Can't Convert Variables to Numerics

QUESTIONS: Can you tell me how I can convert int and factor variables to numeric variables in R? Can you tell me why ...
0
votes
2answers
6k views

Correlation between nominal categorical variables

I have two arrays, whose values are nominal categorical variables. Each element represents a zone of a city: in the first vector we have the class each zone belongs to (so these might also be seen as ...
0
votes
1answer
306 views

Changes in the standard Heatmap plot - symmetric bar colors, show only diagonal values, and column names at x,y axis ticks

I have a heatmap image (correlation between all matrix columns) and I'm straggling to preform all the changes below within the same image: bar colors should be symmetric around zero (e.g., ...
1
vote
2answers
37 views

Which correlation test can be used on low count contigency table?

I need to find the correlation between the ratings given by two parties. Let's say we have 2 coaches, namely A and B. Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "...
0
votes
0answers
12 views

Does the Matthews correlation coefficient (MCC) make sense as a similarity measure between vectors?

I have a set of vectors representing items whose similarity I want to determine. Because the items are given as sets of features, the resulting one-hot encoding vectors are binary, so I am looking for ...
1
vote
1answer
40 views

Is it okay to create a derived model of highly correlated features?

In my data set 2 features C1 and C2 are highly correlated. I did the following steps. Could you please let me know if it is ...
0
votes
2answers
126 views

Building predictive model with low correlated data [closed]

I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
5
votes
2answers
74 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
1
vote
0answers
11 views

R Phi Coefficient Calculation [closed]

I have a dataframe in R and am trying to determine the Phi correlation coefficient between 2 binary (aka dichotomous: 0 or 1) variables, each one in a column (column1 and column2). I have installed ...
0
votes
0answers
10 views

How to determine correlation between ports and ticket prices?

I have titanic dataset, from which I have extracted ticket fares and embarkment ports. I am trying to find out if there is a correlation between embarkment port and ticket fare, so I constructed ...
0
votes
0answers
1k views

Should the type of Boolean categorical features be numerical or categorical after encoding?

There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1. I will apply correalation elimination on ...
0
votes
0answers
15 views

Simple score function with 4 different indicators

I want to create a function, which returns a value between (0,1) or (-1,1). The result of this function is then used for a boolean decision. E.g. if the value is closer to 0 decision ...
4
votes
1answer
88 views

How to find lagged cross correlation between time series?

I have 2 time series, $X$ and $Y$, and I'm trying to find the best lag range that correlates $X$ to $Y$ (find the amount(s) of lag of $X$ that best correlate to the target variable $Y$). For instance, ...
0
votes
1answer
43 views

Finding the worst affected industry due to COVID in terms of unemployment

My goal is to find the worst affected industries from COVID—19 in terms unemployment. In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
0
votes
0answers
134 views

Correlation between continuous variables and multi class categorical variables in python

I was trying to figure out a way of finding a correlation between continuous variables and a non-binary target categorical label. The only thing I though of is by fitting the labels into Multinomial ...
0
votes
0answers
9 views

how to determine the relationship between attributes/whether one has impact the other

I am trying to build a model that determines whether two products have an impact on each other's sales performance. Ideally, the result will provide me a ranking/score between each 2 product pairs. I ...
0
votes
1answer
129 views

Alternative methods for novelty detection and correlations

Hey mates I have the following project: Imagine having two datasets A and B. Each dataset consits of 101 time series with the ...
0
votes
0answers
12 views

How to find the lag that shows highest correlation between two time series variables?

I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
5
votes
2answers
107 views

How to interpret two continous variables output using GAM?

I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (...
0
votes
0answers
30 views

How to measure (and get rid of) biases in training data?

There might be correlation in the training data that do not occur in 'nature' (or occur to a much smaller extent, or maybe we are legally required to not look at these correlations). In my concrete ...
1
vote
1answer
21 views

Removing features that correlate with the target label

I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that ...
0
votes
0answers
37 views

How to quantify the influence of multiple variables on a time series data

I have three variables in my time series data ($X_1,X_2,X_3$). I then calculate $y = (X_1+X_2)-X_3$ for each time $t$. Now, I need to look at a time interval and come up with a quantitative measure ...
0
votes
0answers
10 views

Analysing process data with sub groupings and checking for correlation

I have a dataset of process data for different equipment with many sensors. I would like to check the correlation of the different sensors to see if there is any strong correlation between some ...
0
votes
2answers
92 views

Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
1
vote
0answers
28 views

Finding correlations between plant features?

I have a list of flowers, and each has a color and a fragrance intensity. Each data row has a color and intensity for one breed of flower, like this: ...

1
2 3 4 5
7