Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
132 votes
1 answer
379k views

How to get correlation between two categorical variable and a categorical variable and continuous variable?

I am building a regression model and I need to calculate the below to check for correlations Correlation between 2 Multi level categorical variables Correlation between a Multi level categorical ...
GeorgeOfTheRF's user avatar
16 votes
5 answers
4k views

Beginner math books for Machine Learning

I'm a Computer Science engineer with no background in statistics or advanced math. I'm studying the book Python Machine Learning by Raschka and Mirjalili, but when I tried to understand the math of ...
4 votes
2 answers
127 views

Machine Learning with intended missing values

I have a dataset relating to humans completing reviews, the target variable is whether the review decision is correct / incorrect and one of my features is a trailing 4 week accuracy score for the ...
Stats DUB01's user avatar
17 votes
2 answers
11k views

High-dimensional data: What are useful techniques to know?

Due to various curses of dimensionality, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/...
ASX's user avatar
  • 451
13 votes
2 answers
557 views

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not? Data science seems to seem class imbalance as problematic and needing special techniques ...
Dave's user avatar
  • 3,698
11 votes
3 answers
2k views

Statistics + Computer Science = Data Science? [closed]

i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But ...
user3643160's user avatar
2 votes
4 answers
1k views

Best methods to solve class imbalance problem and why?

I have a data set where I need to detect fraud. 99% are not fraud and 1% are. What methods can be used to solve problems where classes are imbalanced?
John Constantine's user avatar
29 votes
4 answers
3k views

Books about the "Science" in Data Science? [closed]

What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and ...
Anton's user avatar
  • 399
16 votes
1 answer
16k views

How many features to sample using Random Forests

The Wikipedia page which quotes "The Elements of Statistical Learning" says: Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$ features are used in each split. ...
Valentin Calomme's user avatar
7 votes
1 answer
824 views

How to detect overfitting of a stock screener

The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria. Stock Screeners are then "backtested" by simulating the results of applying in ...
Emily Crutcher's user avatar
3 votes
2 answers
6k views

What are the relationships/differences between Bias, Variance and Residuals?

I've been trying to find an answer to this question for a long time. What are the relationships/differences between Bias, Variance and Residuals? I think I do understand Bias, Variance and Residuals ...
Minu's user avatar
  • 795
1 vote
1 answer
112 views

How to analyze repeated measure data for prediction?

In my work, we collect sales data of our products. We have a set of 1st level customers (lets call that group as jacks) with whom we do we business. These jacks then sell our products to end customers ...
The Great's user avatar
  • 2,565
0 votes
1 answer
296 views

How to compare models and which settings to keep constant? [closed]

I already posted this in another forum but no response. So, posting it here. Currently, in clinical practice, clinicians use a score (as a single feature) to predict the mortality of a patient. Now in ...
The Great's user avatar
  • 2,565
0 votes
1 answer
102 views

What is the difference between accuracy in percentage (classification metric) and MPE (regression metric) as evaluation metrices?

What is the difference between accuracy in percentage (classification metric) and MPE (regression metric) as evaluation metrices?
Mehmet Gökdelen's user avatar
60 votes
5 answers
53k views

Neural networks: which cost function to use?

I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "...
daniel451's user avatar
  • 723
26 votes
7 answers
7k views

Is Python a viable language to do statistical analysis in?

I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
confused's user avatar
  • 488
14 votes
3 answers
774 views

When are p-values deceptive?

What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance? Are there specific problem types that fall into this category?
user179's user avatar
  • 143
12 votes
1 answer
9k views

Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67. I had the same algorithms and same set of hyperparamters during yesterday and ...
The Great's user avatar
  • 2,565
11 votes
2 answers
18k views

What is the Time Complexity of Linear Regression?

I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) ...
user134132523's user avatar
9 votes
5 answers
11k views

Why 100% accuracy on test data is not good?

I was asked this question in an interview and wasn’t able to give a satisfactory answer not only upto the interviewers' expectations but of my own as well. The question was as above only, he later ...
Rishabh Sharma's user avatar
7 votes
1 answer
250 views

bias and variance trade off related question

I am having difficulty to understand the expected squared errors formula in this website: $y=f(x)+e$ true regression line $\hat{y}=\hat{f}(x)$ your estimated regression line $error(x)=\bigg(\...
gyambqt's user avatar
  • 71
7 votes
2 answers
33k views

How to find a confidence level given the z value

I have a task from which I would like to find the confidence level given the z value. I have a sample population. From that population, given its distribution, I would like to find the confidence ...
user3841581's user avatar
6 votes
1 answer
562 views

Is Data Science just a trend or is a long term concept? [closed]

I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data ...
Filipe Ferminiano's user avatar
5 votes
1 answer
3k views

Target encoding with KFold cross-validation - how to transform test set?

Let's say I have a categorical feature (cat): ...
Xaume's user avatar
  • 192
4 votes
2 answers
1k views

Anomaly detection in multiple parameters

I am a newbie to data science with a typical problem. I have a data set with metric1, metric2 and metric3. All these metrics are interdependent on each other. I want to detect anomalies in metric3. ...
codejammer's user avatar
3 votes
3 answers
4k views

Root Mean Squared Error (RMSE) - significance of square root

What is the significance of the square root in root-mean-square-error? Essentially, my question is: what is the difference between (rms error) and (rms error)$^2$?
akashrajkn's user avatar
3 votes
2 answers
5k views

VAR model ValueError: x already contains a constant

I'm using VAR model for multivariate time series. The structure is that although each variable is a linear function of past lags of itself and past lags of the other variables, one and/or two of the ...
Abs 's user avatar
  • 181
3 votes
1 answer
940 views

What is the interpretation of the expectation notation in the GAN formulation?

I'm confused about the expectation notation in the context of GAN loss functions. The GAN loss for the discriminator is binary cross-entropy. ie: is this real or not. real = $D(x)$ (ie: give ...
xela's user avatar
  • 31
3 votes
3 answers
8k views

How to convert regression into classification?

So I have a regression problem with bunch of features X, and labels in the amount (price $). How can I convert it to classification problem? I have read about convert label from continuous to ...
Nauman Akram's user avatar
2 votes
1 answer
610 views

difference between feature interactions and confounding variables

Let me define the problem space. I am working a binary classification problem. I am trying to build a causal model as well as predictive model. My aim is to find list of significant features (based ...
The Great's user avatar
  • 2,565
2 votes
1 answer
112 views

Information leakage when train/test are truly i.i.d.?

I am well aware that to avoid information leakage, it is recommended to fit any transformation (e.g., standardization or imputation based on the median value) on the training dataset and applying it ...
thereandhere1's user avatar
2 votes
2 answers
144 views

How do you define the steps to explore the data? [closed]

I'm falling in love with data science and I'm spending a lot of time studying it. It seems that a common data science workflow is: Frame the problem Collect the data Clean the data Work on the data ...
Daniel's user avatar
  • 123
1 vote
1 answer
929 views

What is "oracle" in statistics?

When I read several statistical papers, they mention "oracle" property or "oracle" estimator. What do they mean by "oracle"? I understand this oracle is not a company name, but have no idea what this ...
Roy_Oishi's user avatar
1 vote
0 answers
118 views

Normalizing and joining of independent logistic regression model's prediction

I need to train several Logistic regression models on a different set of data (with a different set of labels): ...
United121's user avatar
  • 131
1 vote
1 answer
1k views

How do I convert an L2 norm to a probability?

I am using the dot product as a way to measure the similarly of two facial-model vectors extracted by a ML algorithm (OpenFace in fact). I would like to convert the L2 norm to a probability U[0,1] in ...
jtlz2's user avatar
  • 151
1 vote
3 answers
691 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
Rohit Gavval's user avatar
1 vote
2 answers
365 views

How to build a model on a dataset having 40% missing values in most of the variables?

I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category)...
Max64's user avatar
  • 31
0 votes
0 answers
96 views

What are the pros and cons of R2 (coefficient of determination)?

What are the pros and cons of $R^2$ (coefficient of determination) which is an evaluation metric?
Mehmet Gökdelen's user avatar
0 votes
1 answer
71 views

Comparing two groups at large scale

Let's consider we have two datasets. Dataset "A" and Dataset "B". Dataset "A" has two columns. Supplier_id and "Status" (pass and fail are values for status ...
The Great's user avatar
  • 2,565
0 votes
1 answer
91 views

How can I use data science / machine learning / statistics / data analytics techniques to find better customers in the future? [closed]

I work for a business to business company with a data set containing approximately 2700 businesses as customers. For each business customer I have approx number of staff, location, industry, the sales ...
marcommunication's user avatar