Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
132 votes
1 answer
378k views

How to get correlation between two categorical variable and a categorical variable and continuous variable?

I am building a regression model and I need to calculate the below to check for correlations Correlation between 2 Multi level categorical variables Correlation between a Multi level categorical ...
GeorgeOfTheRF's user avatar
60 votes
5 answers
53k views

Neural networks: which cost function to use?

I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "...
daniel451's user avatar
  • 723
46 votes
12 answers
54k views

Data Science in C (or C++)

I'm an R language programmer. I'm also in the group of people who are considered Data Scientists but who come from academic disciplines other than CS. This works ...
Hack-R's user avatar
  • 1,919
38 votes
3 answers
193k views

Calculation and Visualization of Correlation Matrix with Pandas

I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of ...
gdlm's user avatar
  • 535
29 votes
10 answers
28k views

Any Online R console?

I am looking for an online console for the language R. Like I write the code and the server should execute and provide me with the output. Similar to the website Datacamp.
Gotham's user avatar
  • 291
29 votes
4 answers
3k views

Books about the "Science" in Data Science? [closed]

What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and ...
Anton's user avatar
  • 399
26 votes
7 answers
7k views

Is Python a viable language to do statistical analysis in?

I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
confused's user avatar
  • 488
23 votes
4 answers
739 views

What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data

I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on ...
Peter Kirby's user avatar
20 votes
3 answers
18k views

Overfitting in Linear Regression

I'm just getting started with machine learning and I have trouble understanding how overfitting can happen in a linear regression model. Considering we use only 2 feature variables to train a model, ...
Sachin Krishna's user avatar
17 votes
2 answers
11k views

High-dimensional data: What are useful techniques to know?

Due to various curses of dimensionality, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/...
ASX's user avatar
  • 451
16 votes
1 answer
16k views

How many features to sample using Random Forests

The Wikipedia page which quotes "The Elements of Statistical Learning" says: Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$ features are used in each split. ...
Valentin Calomme's user avatar
16 votes
5 answers
4k views

Beginner math books for Machine Learning

I'm a Computer Science engineer with no background in statistics or advanced math. I'm studying the book Python Machine Learning by Raschka and Mirjalili, but when I tried to understand the math of ...
15 votes
4 answers
6k views

How to specify important attributes?

Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to ...
vefthym's user avatar
  • 503
15 votes
2 answers
5k views

Analyzing A/B test results which are not normally distributed, using independent t-test

I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution. In fact the distribution resembles more closely the Landau Distribution. I ...
teebszet's user avatar
  • 253
14 votes
3 answers
773 views

When are p-values deceptive?

What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance? Are there specific problem types that fall into this category?
user179's user avatar
  • 143
13 votes
6 answers
824 views

Datasets understanding best practices

I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean ...
Jack Twain's user avatar
13 votes
2 answers
555 views

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not? Data science seems to seem class imbalance as problematic and needing special techniques ...
Dave's user avatar
  • 3,698
12 votes
4 answers
16k views

Is GLM a statistical or machine learning model?

I thought that generalized linear model (GLM) would be considered a statistical model, but a friend told me that some papers classify it as a machine learning technique. Which one is true (or more ...
user77571's user avatar
  • 313
12 votes
1 answer
9k views

Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67. I had the same algorithms and same set of hyperparamters during yesterday and ...
The Great's user avatar
  • 2,565
12 votes
1 answer
415 views

ngram and RNN prediction rate wrt word index

I tried to plot the rate of correct predictions (for the top 1 shortlist) with relation to the word's position in sentence : I was expecting to see a plateau sooner on the ngram setup since it ...
Arkantus's user avatar
  • 157
11 votes
3 answers
8k views

Best languages for scientific computing [closed]

It seems as though most languages have some number of scientific computing libraries available. Python has Scipy Rust has <...
ragingSloth's user avatar
  • 1,824
11 votes
3 answers
93k views

How to group identical values and count their frequency in Python?

Newbie to analytics with Python so please be gentle :-) I couldn't find the answer to this question - apologies if it is already answered elsewhere in a different format. I have a dataset of ...
new_analyst's user avatar
11 votes
3 answers
2k views

Data Science oriented dataset/research question for Statistics MSc thesis

I'd like to explore 'data science'. The term seems a little vague to me, but I expect it to require: machine learning (rather than traditional statistics); a large enough dataset that you have to run ...
user3279453's user avatar
11 votes
3 answers
2k views

Statistics + Computer Science = Data Science? [closed]

i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But ...
user3643160's user avatar
11 votes
2 answers
9k views

Why use bootstrapping?

The wiki page for bootstrapping says that you use it in the case where the underlying distribution is unknown. Why is bootstrapping, or sampling with replacement, better than just calculating the ...
sebastianspiegel's user avatar
11 votes
2 answers
18k views

What is the Time Complexity of Linear Regression?

I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) ...
user134132523's user avatar
11 votes
3 answers
12k views

Relationship between KS, AUROC, and Gini

Common model validation statistics like the Kolmogorov–Smirnov test (KS), AUROC, and Gini coefficient are all functionally related. However, my question has to do with proving how these are all ...
Steven's user avatar
  • 111
10 votes
3 answers
350 views

How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
Bridgeburners's user avatar
10 votes
1 answer
639 views

A clear visualization of a two-way ANOVA

To provide a full yet simple picture of a 3-level, one-way ANOVA, I use the following visualization where variation within each group (the filled circles) and variation between the groups (black ...
Reza Norouzian's user avatar
9 votes
6 answers
6k views

When to use mean vs median

I'm new to data science and stats, so this might seems like a beginner question. I'm working on a dataset where I've user's Twitter followers gain per day. I want to measure the average growth he had ...
Mukul Jain's user avatar
9 votes
2 answers
7k views

What is the rationale for discretization of continuous features and when should it be done?

Continous feature discretization usually leads to lose of information due to the binning process. However most of the Top solutions for Kaggle Titanic are based on discretization(age,fare). When ...
drichlet's user avatar
9 votes
2 answers
225 views

Ways to reconstruct shuffled pixels of a video file?

Suppose that you have a video file which pixel order has been shuffled once. That is, a random order have been defined once and applied to all frames. Does it exist some known approach for ...
Denis Dollfus's user avatar
9 votes
5 answers
11k views

Why 100% accuracy on test data is not good?

I was asked this question in an interview and wasn’t able to give a satisfactory answer not only upto the interviewers' expectations but of my own as well. The question was as above only, he later ...
Rishabh Sharma's user avatar
8 votes
3 answers
6k views

How to find out if two datasets are close to each other?

I have the following three datasets. ...
Kartikeya Sharma's user avatar
8 votes
3 answers
8k views

Modality of data

Can anyone please explain in clear words what is generally meant by "modality of data"? I know what modality means with respect to distributions.
Julia's user avatar
  • 81
8 votes
2 answers
220 views

Linearly increasing data with manual reset

I have a linearly increasing time series dataset of a sensor, with value ranges between 50 and 150. I've implemented a Simple Linear Regression algorithm to fit a regression line on such data, and I'm ...
ArunDhaJ's user avatar
  • 183
8 votes
1 answer
46k views

Am I doing a log transformation of data correctly?

I'm doing some exploratory data analysis on some data and I get these histograms: That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the ...
Jon's user avatar
  • 183
8 votes
2 answers
287 views

Evaluating Recommendation engines

What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare ...
Sreejithc321's user avatar
  • 1,920
8 votes
2 answers
8k views

Covariance as inner product

Why is covariance considered as inner product if there is no projection of one vector onto another? Right now I perceive this as just a multiplication of $x$ segment of vector($x_i - \bar{x}$) and $y$...
user641597's user avatar
8 votes
2 answers
2k views

A/B testing: How to calculate p-value on post test segments?

My question on A/B testing is about doing post test segmentation analysis. For example: I run an A/B test on my website to track bounce rate. On the treatment group, i put a video to explain ...
jxn's user avatar
  • 233
7 votes
2 answers
286 views

Data Science as a Social Scientist?

as I am very interested in programming and statistics, Data Science seems like a great career path to me - I like both fields and would like to combine them. Unfortunately, I have studied political ...
Christian Sauer's user avatar
7 votes
1 answer
821 views

How to detect overfitting of a stock screener

The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria. Stock Screeners are then "backtested" by simulating the results of applying in ...
Emily Crutcher's user avatar
7 votes
1 answer
1k views

How to numerically estimate MLE estimators in python when gradients are very small far from the optimal solution?

I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables. Something like N ~ (f(x), g(x)). I generate a ...
Rodrigo Stv's user avatar
7 votes
5 answers
238 views

Reducing the effect of down voters with rating system

I have a site in which users rate things in a 1-5 star system. Once an item reaches the top of the charts, some users tend to start rating it 1 star even though it got a majority of 4-5 stars to get ...
ParoX's user avatar
  • 121
7 votes
3 answers
12k views

Testing independence of random variables in Python

Are there any tools available in Python that allow for testing of independence of two random variables (data columns)? I have two columns of data $X$ and $Y$. They can be both discrete, with values $\{...
WoofDoggy's user avatar
  • 343
7 votes
2 answers
126 views

Computational aspects are typically ignored by statisticians

In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that : Although data science can be seen as a continuation of statistics, the ...
Koinos's user avatar
  • 141
7 votes
2 answers
33k views

How to find a confidence level given the z value

I have a task from which I would like to find the confidence level given the z value. I have a sample population. From that population, given its distribution, I would like to find the confidence ...
user3841581's user avatar
7 votes
2 answers
244 views

Which statistical test tells which classifier performs better than the other?

I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do ...
girl101's user avatar
  • 1,161
7 votes
2 answers
2k views

IID violation in machine learning

Imagine I'm collecting some training data. Lets say I collect a 30minute time series from 1000 people so I have 1000 observations (rows) in my feature matrix. I train some model (lets say a neural ...
Simon's user avatar
  • 1,071
7 votes
2 answers
487 views

When is the sum of models the model of the sum?

The response variable in a regression problem, $Y$, is modeled using a data matrix $X$. In notation, this means: $Y$ ~ $X$ However, $Y$ can be separated out into different components that can be ...
Sam Castillo's user avatar

1
2 3 4 5
23