Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

40 questions
Filter by
Sorted by
Tagged with
379k views

How to get correlation between two categorical variable and a categorical variable and continuous variable?

I am building a regression model and I need to calculate the below to check for correlations Correlation between 2 Multi level categorical variables Correlation between a Multi level categorical ...
• 2,028
4k views

Beginner math books for Machine Learning

I'm a Computer Science engineer with no background in statistics or advanced math. I'm studying the book Python Machine Learning by Raschka and Mirjalili, but when I tried to understand the math of ...
127 views

Machine Learning with intended missing values

I have a dataset relating to humans completing reviews, the target variable is whether the review decision is correct / incorrect and one of my features is a trailing 4 week accuracy score for the ...
• 161
11k views

High-dimensional data: What are useful techniques to know?

Due to various curses of dimensionality, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/...
• 451
557 views

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not? Data science seems to seem class imbalance as problematic and needing special techniques ...
• 3,698
2k views

Statistics + Computer Science = Data Science? [closed]

i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But ...
• 163
1k views

Best methods to solve class imbalance problem and why?

I have a data set where I need to detect fraud. 99% are not fraud and 1% are. What methods can be used to solve problems where classes are imbalanced?
3k views

Books about the "Science" in Data Science? [closed]

What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and ...
• 399
16k views

How many features to sample using Random Forests

The Wikipedia page which quotes "The Elements of Statistical Learning" says: Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$ features are used in each split. ...
• 5,986
824 views

How to detect overfitting of a stock screener

The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria. Stock Screeners are then "backtested" by simulating the results of applying in ...
6k views

What are the relationships/differences between Bias, Variance and Residuals?

I've been trying to find an answer to this question for a long time. What are the relationships/differences between Bias, Variance and Residuals? I think I do understand Bias, Variance and Residuals ...
• 795
1 vote
112 views

How to analyze repeated measure data for prediction?

In my work, we collect sales data of our products. We have a set of 1st level customers (lets call that group as jacks) with whom we do we business. These jacks then sell our products to end customers ...
• 2,565
296 views

How to compare models and which settings to keep constant? [closed]

I already posted this in another forum but no response. So, posting it here. Currently, in clinical practice, clinicians use a score (as a single feature) to predict the mortality of a patient. Now in ...
• 2,565
102 views

What is the difference between accuracy in percentage (classification metric) and MPE (regression metric) as evaluation metrices?

What is the difference between accuracy in percentage (classification metric) and MPE (regression metric) as evaluation metrices?
53k views

Neural networks: which cost function to use?

I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "...
• 723
7k views

Is Python a viable language to do statistical analysis in?

I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
• 488
774 views

When are p-values deceptive?

What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance? Are there specific problem types that fall into this category?
• 143
9k views

Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67. I had the same algorithms and same set of hyperparamters during yesterday and ...
• 2,565
18k views

What is the Time Complexity of Linear Regression?

I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) ...
11k views

Why 100% accuracy on test data is not good?

I was asked this question in an interview and wasn’t able to give a satisfactory answer not only upto the interviewers' expectations but of my own as well. The question was as above only, he later ...
250 views

• 101
610 views

difference between feature interactions and confounding variables

Let me define the problem space. I am working a binary classification problem. I am trying to build a causal model as well as predictive model. My aim is to find list of significant features (based ...
• 2,565
112 views

Information leakage when train/test are truly i.i.d.?

I am well aware that to avoid information leakage, it is recommended to fit any transformation (e.g., standardization or imputation based on the median value) on the training dataset and applying it ...
144 views

How do you define the steps to explore the data? [closed]

I'm falling in love with data science and I'm spending a lot of time studying it. It seems that a common data science workflow is: Frame the problem Collect the data Clean the data Work on the data ...
• 123
1 vote
929 views

What is "oracle" in statistics?

When I read several statistical papers, they mention "oracle" property or "oracle" estimator. What do they mean by "oracle"? I understand this oracle is not a company name, but have no idea what this ...
1 vote
118 views

Normalizing and joining of independent logistic regression model's prediction

I need to train several Logistic regression models on a different set of data (with a different set of labels): ...
• 131
1 vote
1k views

How do I convert an L2 norm to a probability?

I am using the dot product as a way to measure the similarly of two facial-model vectors extracted by a ML algorithm (OpenFace in fact). I would like to convert the L2 norm to a probability U[0,1] in ...
• 151
1 vote
691 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
• 321
1 vote
365 views

How to build a model on a dataset having 40% missing values in most of the variables?

I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category)...
• 31
96 views

What are the pros and cons of R2 (coefficient of determination)?

What are the pros and cons of $R^2$ (coefficient of determination) which is an evaluation metric?