Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

0
votes
3answers
44 views

How to statistically prove that a column in a dataframe is not needed

I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
-1
votes
0answers
29 views

Linear Regression performance in R

I have a problem with R's performance: here is my Script. The problem is that I need to use it in a base with ~6m of clients (with one linear model per customer) and it's taking to long to process. ...
1
vote
1answer
22 views

Can anyone explain me the difference between Factor Anaysis and PCA?

Is PCA and Factor Analysis same? Both are used for Data dimension reduction but theoretically I am not able to find the difference between them? I did FA in SPSS to reduce number of variables in my ...
0
votes
1answer
15 views

PCA-like analysis for dataset that has both categorical and continuous variables

I have a dataset containing a categorical variable and multiple continuous variables. The categorical variables are coded as discrete integers, whereas the continuous variables are just a range of ...
1
vote
2answers
41 views

Statistical test for machine learning

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to ...
3
votes
0answers
32 views

Which background is the best for Data Science? [closed]

From what I'm aware most people working on data science these days come from very different backgrounds (e.g. statistics, engineering, computer science, economics). Since there weren't any BSc or MSc ...
3
votes
2answers
13 views

Regression or Correlation for this RQ?

Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in ...
3
votes
2answers
25 views

Why isn't local averaging (including KNN) used often for regression?

My professor said that the "holy grail of regression" is the function E(Y|X=x) i.e. the conditional expectation of Y on X. In practice, you'd take a small window of X and take the average value of Y ...
2
votes
1answer
26 views

VAR model ValueError: x already contains a constant

I'm using VAR model for multivariate time series. The structure is that although each variable is a linear function of past lags of itself and past lags of the other variables, one and/or two of the ...
2
votes
3answers
29 views

In machine learning how to find feature interdepencies? [closed]

Given a data set of N features, wherein some the features in this set were derived from other features from the same set, I am trying to discover inter dependencies between features (something like ...
0
votes
1answer
9 views

What Does the Normalization Factor Mean in the AdaBoost Algorithm?

I am studying the AdaBoost algorithm. The update rule for a weak hypothesis is: $Dt+1(i) = Dt(i)exp(−αtyiht(xi))/zt $ where $zt$ is a normalization factor chosen so that $Dt+1$ is a distribution. ...
1
vote
1answer
8 views

Metrics/Methods for deciding duration of video retention for on-demand websites

This might be a general question but I thought this might be the best place to brainstorm. If I had a video website that only wants to retain videos on-demand for a certain number of days before ...
0
votes
0answers
25 views

Why use parametric test at all if non parametric tests are 'less strict'

I have read from several sources, even in my undergrad courses, that parametric tests require the data to have a certain distribution, for instance normal, whilst non-parametric don't. I have ...
1
vote
1answer
58 views

How often do we see normally distributed data

I am having difficulty in exactly understanding several statistical tests, such as the t-test and ANOVA test. These tests require that the data we use be normally distributed. However, whilst ...
1
vote
1answer
32 views

How to add noise to supervised (binary-classifier)?

Note: The question is not about validating/testing a trained model. Say i have an unlabeled features set, I want to approximate the true labels (for the sake of argument lets assume it's a binary ...
0
votes
0answers
10 views

Need help with a problem statement in separating poisson processes with an amplitude

Suppose you have N Poisson like processes (with an amplitude) Is there a technique to separate these signals out - so we finally get a signal classification of each point?
0
votes
0answers
26 views

fit distribution to the data

I want to fit a distribution that is as close as possible to my data. The data is regarding the number of events that arrive in a time period. In theory, Poisson distribution is used for similar cases,...
2
votes
1answer
27 views

Knowledge of statistics To be a data scientist [closed]

I am an undergraduate student pursuing my BSc in Maths. I have taken statistics classes in first year. I want to know how much statistics do I need to know to get a decent place in the data science ...
1
vote
0answers
50 views

How to plot a heatmap-like plot for categorical features?

I would greatly appreciate let me know how to plot a heatmap-like plot for categorical features? In fact, based on this post, the association between categorical ...
3
votes
0answers
25 views

How to best estimate the coefficients of a confusion matrix in case of strong class imbalance?

I have a trained binary classifier (forget about how this was trained and think of it as a magical black box) and I would like to measure its classification performance (e.g. compute a confusion ...
1
vote
1answer
39 views

Best statistic books which have solutions

I saw some questions which are asked about books in data science, so I think my question is not out of topic. I am looking for a book in statistic which has complete answers for exercises. The ...
3
votes
1answer
19 views

Mixture Density Network: determine the parameters of each Gaussian component

I am reading Bishop's Mixture Density Network paper at: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-ncrg-94-004.pdf This is a good paper, but I am still confused about ...
0
votes
0answers
12 views

Binomial Test with Bonferroni Correction

I have 28 images that I am showing individuals (2 at a time) in which they select their favorite image. I have the following metrics for each image image1,image2,number of people who selected image1,...
3
votes
1answer
22 views

Should we identify outliers from population prior to taking sample?

I am revising undergrad statistics course via this course, where i am learning techniques to pull out sample from population. While ensuring that sample is a decent representative of the population, ...
3
votes
1answer
52 views

What's a reasonable distribution to model views over time of… this question?

Sorry for the self-referential title ;) I'd be curious to know what's a curve that could be used to model - in the scale of (say) weeks* - the "engagement lifecycle" of a single news page of an ...
1
vote
0answers
25 views

Statistical Significance in Pairwise Ranking Algorithm

Can anyone recommend an algorithm/toolkit to rank items that have been rated in a hot-or-not style that gives statistical significance? For example, out of a set of N images, two images are shown to ...
1
vote
0answers
16 views

Statistical Significance for Pairwise Ranking Algorithm

I am using this library: https://github.com/lucasmaystre/choix/blob/master/notebooks/intro-pairwise.ipynb to rank items that have been rated in a hot-or-not style. For example, two images are shown ...
2
votes
1answer
89 views

Why is sampling useful in machine learning?

I have met that question online and I wanted to know where sampling can simulate complex processes and why? Why is sampling useful in machine learning? Sampling can increase the accuracy of the ...
1
vote
1answer
11 views

How the term “R-squared” in VIF(variance inflation factor) is different from normal R-squared calculation?

In normal calculation of R2 , more the value of R2 , it indicates variable represents more variance across the dataset. But in the calculation of VIF (variance inflation factor), higher the value of ...
0
votes
0answers
15 views

Can someone explain in layman terms how the Contrastive Divergence algorithm works step by step?

I am interested in learning about Restricted Boltzmann Machines (RBMs), but I have trouble understanding how an RBM is trained using contrastive divergence. There are only few papers on this topic and ...
0
votes
1answer
28 views

predict future sale from past 3 month data

I have the 3-month sale(branch wise) and I want to predict next month sale. There are huge differences between total sale in between months. The simple average of the month's to predict next month ...
0
votes
0answers
10 views

Looking for correlation between a big set of variables: divided in groups or all together?

I have a tiny classified dataset (2 classes and over 20 samples for each class) where I would like to find the highest pair-correlated variables. I doubt if I should mix both groups of samples and run ...
0
votes
2answers
33 views

Best ML model for predicting yearly data with many blocks?

I would like to forewarn by saying I am by no means an expert in this topic and I apologize if I mix terminology, phrased the question wrong, or any information is incorrect. I would like to use a ...
1
vote
0answers
24 views

How many repititions of a stochastic algorithm should be done before the results can be considered trustworthy?

I have written a stochastic machine learning algorithm. Each time I run the algorithm on the same test data I get different results. I determine accuracy each time compared to ground truth. In order ...
2
votes
1answer
127 views

What is the Time Complexity of Linear Regression?

I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) ...
2
votes
2answers
69 views

Testing independence of random variables in Python

Are there any tools available in Python that allow for testing of independence of two random variables (data columns)? I have two columns of data $X$ and $Y$. They can be both discrete, with values $\{...
0
votes
0answers
20 views

Differentiating distributions algorithmically/programmatically

Let's say I have a distribution and I am trying to see what kind it most closely follows (normal, Poisson, etc). Here is a discussion that may be somewhat related: Detecting Different Distributions ...
0
votes
0answers
16 views

Semi-supervised Learning doubt

I'm reading "Hands on machine learning" by Aurelien Geron. He stated that semi-supervised learning is: Some photo-hosting services, such as Google Photos, are good examples of this. Once you ...
6
votes
2answers
109 views

Computational aspects are typically ignored by statisticians

In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that : Although data science can be seen as a continuation of statistics, the ...
1
vote
1answer
28 views

Why should re-sampling change the value of model's coefficients?

I have the code below in python to create LinearRegression model. When I train the model with re-sampled data, I get different values for its coefficients. I can't understand why that happens. Can you ...
0
votes
1answer
19 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished the race in 45 seconds,35 ...
0
votes
0answers
13 views

What is the variance of k-fold cross validation

My course teacher write this in his notes but I cannot understand: Conduct k-fold cross validation n times with $k=n$, each time randomly splitting data sets into folds. $d_i^j$ measures the ...
1
vote
1answer
41 views

What Introductory Statistics book would you recommend? [closed]

Is there a particularly good book or good publisher for teaching Statistics from beginner level for someone interested in studying Data Science?
1
vote
0answers
46 views

What is the relationship between correlation ratio and one-way Anova?

According to the answer to this post, it is recommended to use one-way anova to compute the dependence between a categorical and a numerical variable. Besides, the second answer to this post says ...
1
vote
2answers
48 views

Best programming language [closed]

I have a masters in pure statistics. I have no knowledge of any programming language. Which is the best programming language that I should learn about and will help enhance my career in data analysis. ...
0
votes
0answers
9 views

Training a classifier on 2 catalogs combined

I'm currently trying to train a machine to identify flashes of light in the sky as supernova (SN) or not. To do this, I'm combining 2 different catalogs: A big catalog (99% SN) with brighter flashes ...
0
votes
0answers
14 views

Build machine learning model with all binary predictor variables

I have data with 250 predictor variables, all of them are binary variables (0=Absent, 1 =Present) and my Y variables is probability (Y=1 means highest chance of occurrence of an event and Y=0 means ...
0
votes
0answers
9 views

Which method can I use to explore dependences between categorial(nominative) variables?

There is some sociological research data. For example: How can I explore some sort of correlation between income level and district, or which statistical measures based on this data can I calculate? ...
0
votes
1answer
26 views

How to evaluate RMSE with standard deviation

I have regression model, where target is between 0 to 1. standard deviation of target is 0.817 and RMSE of model on hold out is 0.52. I am wondering if this good model or not. Any feedback will be ...
1
vote
1answer
48 views

Churn definition for non-contractual services

I am computing churn definition i.e, the number of days after which we will say a customer has churned in fashion retail, etc. Currently, I am using the transaction dates to get the average days ...