# Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

1,037 questions
Filter by
Sorted by
Tagged with
4 views

### Pedagogical examples relating to neural networks and statistics (distributions) of inputs and outputs?

The way I was taught and commonly learn about various neural network technologies is by looking at the problem from the "specific example" stand point. For example, if I were to research ...
• 123
12 views

### Clustering with missing (incomplete) data by design

I'm a psychology PhD student and trying to formulate a research plan. As an example, let's say I want to know which subjects are "cost efficient" to study, by clustering subjects. I'd have a ...
1 vote
17 views

### How to test the confidence for a rule based system?

I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole ...
3 views

### I do not understand normalization and standardization

in my ML Course at Uni, a big topic was normalization and standardization, but still - I don't really get why we do that. More specifically, I'm working on a (fairly complex) CNN to predict missing ...
1 vote
17 views

### What methods I could use to analyze the contingency table?

I am data science beginner, and I have a question about methods that I could use to analyze the following data. It is a simple case, I am trying to check the influence of cohabitation before marriage ...
• 11
20 views

### How to Calculate values inside DBLP collaboration un-directed network? | Using python

(EDITED- thanks) I have loaded txt file (com-dblp.ungraph.txt) which includes network to a COLAB notebook. I want to calculate describe vals of the net in the COLAB- how can I do it? Number of nodes ...
10 views

### Fit Regression Model: Variables Dependent by Interval

I want to use the "Fit Regression Model" tool in Minitab with a series of predictors and one response. I am trying to reduce the number of variables I use as predictors. There are three ...
7 views

### Estimating effect of coefficient in log-linear model

I'm working on a log-linear model which has multiple variables with their coefficients. The model estimates a variable log(Y) ~ X. I'm trying to estimate the effect ...
• 121
10 views

### Capacity planning and modelling

I have a business case in which I am going to model how many devices are required given the predicted workload in a series of monthly cohorts in the next ten years. The work could come from multiple ...
• 763
17 views

### Kullback Leibler divergence of two datasets having different sizes

is there a way to apply KL on 2 datasets of different sizes? ps: my datasets are 2 heat maps with 120x180 and 60x90 as width and height respectively.
• 23
16 views

### Anomaly Detection in a time series data using statistical methods

0: I've been working on time series data recently and need to detect the anomalies observed using HoltWinters(Triple exponential smoothing) model. Base: I implemented Triple exponential smoothing, ...
7 views

### Split new values based on historical data - best methods

I have daily hitorical data, something like this (Mon - 100, Tue - 100, Wed - 70, Thu - 110, Fri - 200, Sat - 100, Sunday - 30). I have several values for the same historical period. Next I have a new ...
13 views

### for which Machine learning models we should Discrete continuous features?

my question may be seems duplicate but its because that I couldn't find any clear and unequivocal answer for this question on the web. 1. I want to know that when should I discrete and categorize ...
• 3
12 views

### How do I compare value counts in each year when each year has a different sample size?

Let's say I went out each year asking people whether they really liked Pepsi, did not care about it, or hated it. But in the first year I asked people at my company when it was small, then the second ...
9 views

### Metrics for changes in datapoints

I am looking for a metric which quantifies the noise in data with respect to neighbouring data points. I have time series data and I want to check how much values change from a previous to a following ...
22 views

### Is it a right way to score the importance of categorical features against categorical targets using ANOVA?

I know that f_classif method is mostly used for scoring numerical features and categorical targets. But I'm wondering if it's possible to score features coded by integers or binary codes against ...
• 101
31 views

### How to generate a rule-based system based on binary data?

I have a dataset where each row is a sample and each column is a binary variable. The meaning of $X_{i, j} = 1$ is that we've seen feature $j$ for sample $i$. $X_{i, j} = 0$ means that we haven't seen ...
9 views

### why zero centring of data from activation function is good for deep nural network?

I was reading an article that mentioned reasons why tanh is better than sigmoid and one reason was that tanh gives zero-centered data but I couldn't understand why and how it will affect our network. ...
19 views

### uncertainties in non-convex optimization problems (neural networks)

How do you treat statistical uncertainties coming from non-convex optimization problems? More specifically, suppose you have a neural network. It is well known that the loss is not convex; the ...
• 3
17 views

### Polynomial regression with two variables. How can I find expressions to describe the coefficients?

I'm not sure if this is an appropriate place for this question, so please feel free to redirect me if it is not. I just moved it from Super User, where it seemed like there weren't many similar ...
• 101
10 views

### Recommendations for modelling panel data

sending positive wishes to y'all. I have about 10 years of growth rates in real estate prices and some other macroeconomic variables such as inflation, unemployment rates, fuel prices, growth in ...
• 101
38 views

### What are the requirements for a word list to be used for Bayesian inference?

Intro I need an input file of 5 letter English words to train my Bayesian model to infer the stochastic dependency between each position. For instance, is the probability of a letter at the position 5 ...
5 views

### How would I check the validity of covariates in my linear model on several hundred datasets?

I have this linear model with predictors that I need to prove are statistically significant and pass the necessary lm assumptions. I know for a single dataset, I can use various LM tests, but the ...
17 views

### Interpreting interaction term coefficient in GLM/regression

I'm a psychology student and trying come up with a research plan involving GLM. I'm thinking about adding an interaction term in the analysis but I'm unsure about the interpretation of it. To make ...
23 views

### what degree of fredom one should use while calculating standard diviation for standardizing data

I am writing a function to standardize the data and I found out that we can choose either ddof = 0 or ddof = 1, so I got confused that which one to choose and why? Does this make any difference?
27 views

### Feature importance of a linear regression

What is the easiest and easy to explain feature importance calculation for linear regression? I know I can use Shap to compute feature importance, but I find it difficult to explain it to stakeholders,...
• 209
15 views

### Can we use an independent t-test as a metric for feature importance?

I have a supervised binary classification problem. I tuned an xgboost model on the training set and achieved a reasonably high accuracy on the test set. Now I want to interpret the results of the ...
• 165
11 views

### List of main statistics models

I am not able to find some list of main statistics models. Is is possible to devide statistics models into categories as supervised (regression,classification) x unsupervised (clustering) or is it ...
4 views

### Fisher's exact test when resampling from the population

I am using different ML classifiers for making predictions related to a binary classification. I would like to compare two groups. Let's call them A and B. The already trained and tested classifier ...
• 101
15 views

### How to do binning in matrix data

I have some data like ...
19 views

### How to interpret arg min in the the following equation?

I am studying the following equation: $\hat{s}_m(n) = \text{arg}\text{min}_{s_m(n)\in A_s}|\frac{\psi_m^H}{||\psi_m^H||^2}y_m(n)-s_m(n)|^2$----(1) here $A_s$ is 1x$N$ vector of QPSK symbols, $s_m(n)$ ...
1 vote
46 views

### Incorrect example of applying Bayes theorem

I have been reading the book "The Data Science Design Manual" (by Steven S. Skiena) and I came across an example that explained how the Bayes theorem can be applied that confused me and made ...
• 111
13 views

### How to find an anomalous matrix among many?

Let's say we have a bunch of matrices that we know are non-anomalous. We now receive a new matrix and want to know if it belongs into the group or is way off. Is there a way to do that? I'm thinking ...
34 views

### Is there a Solution Manual for Introduction to Statistical Pattern Recognition Book by Keinosuke Fukunaga?

I'm looking for a solution manual for Introduction to Statistical Pattern Recognition Book by Keinosuke Fukunaga ? I couldn't find anything on the internet . If you have any solution manual by ...
9 views

### Statistically increase mean difference between two data sets?

A dataset will be used to train a binary classification model. For better understanding/visualization, the data set was divided into 2: one set with all the rows that result in prediction value of 1 ...
1 vote
43 views

### What's the purpose of statistical analysis ( statistically important features) vs feature elimination in machine learning

I am developing a classification model for covid19 symptoms (after being ill) and I don't understand statistical analysis importance (some parts of it) 1 Firstly: Basically we perform statystical ...
• 113
33 views

### Comparing performances of different models using hypothesis testing

A common workflow in applied ML is you train different models on some data and evaluate it on a particular test set. I have often seen people just selecting a ML metric based on their requirements and ...
• 63
10 views

### what is meant by walk square in a maximum likelihood estimate

I have some data about maximum likelihood estimates. We have fields of estimate, standard error, pr>chiSq and Walk Square. I have no idea what the Walk Square is referring to - it feels like some ...
• 1
1 vote
11 views

### How should I write the conclusion when doing the same hypothesis on two different subjects and get different results?

I am doing a research trying to find if there's a correlation between twitter sentiments and sales and I'm doing it on two different companies. They are in the same industry and direct competitors. ...
24 views

### How to sample a dataframe or numpy array with a particular interval?

I have the following dataframe : ...
• 1
13 views

### Analyzing series of events

noob in data science here. I would like some advice on the following: I have a series of events that can happen at random times of a day, there is the event start/end time. If an event started already ...
9 views

### Maximum entropy optimization for approximating image quality score distribution - as in Google's "Neural Image Assessment" paper

I am asking this question after a thorough research on the internet and having read every single detail of "NIMA: Neural Image Assessment" by Hossein Talebi and Peyman Milanfar. Before ...
1 vote
11 views

### P-value using Gaussian Discriminate Analysis

I was wondering , in Gaussian Discriminate Analysis (GDA) model, say we have two classes to classify y=0 and y=1 So after fitting the Gaussian over y=0 and y=1 dataset, when we try to predict class ...
10 views

### Filling NaN values

According to my knowledge, before filling nan values we have to check whether data is missing because of MCAR, MAR or MNAR and it depends on how features are correlated with each other and then make a ...
24 views

### Is this a successful implementation of KL Divergence from scratch, and how can I graph all distributions?

I'm attempting to implement a KL Divergence on two imaginary dice from scratch and I'm not sure if it is correct. By 'how do I graph all distributions' I'm wondering if the KL Divergence just a number ...
• 157
6 views

### Model transfer with limit to none label information

I have this problem I hope to get some help here. Say I have a type of product A whose measurements are X_A and an outcome property is ...
• 21
1 vote
35 views

### Using F_regression to find the best significant features

We are trying to use SelectKBest F_Regression scoring function on a pool of 1000 numerical features, and solve a regression problem. Also, we wanted to paralellize the execution of SelectKBest and we ...
10 views

### How to test if a curve is well described by an ellipse?

I have a set of data points in 2D, and I am trying to come up with some sort of statistical to determine if the points fall along an ellipse. My idea so far is to fit an ellipse to the points, take ...
26 views

### When linear regression is really a good model to be used?

By definition of linear regression model: If we note $Y$ the real random variable to be explained (endogenous variable, dependent or response) and $X$ the explanatory variable or fixed effect (...
• 101
1 vote