Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Pedagogical examples relating to neural networks and statistics (distributions) of inputs and outputs?

The way I was taught and commonly learn about various neural network technologies is by looking at the problem from the "specific example" stand point. For example, if I were to research ...
user avatar
  • 123
0 votes
0 answers
12 views

Clustering with missing (incomplete) data by design

I'm a psychology PhD student and trying to formulate a research plan. As an example, let's say I want to know which subjects are "cost efficient" to study, by clustering subjects. I'd have a ...
user avatar
1 vote
1 answer
17 views

How to test the confidence for a rule based system?

I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole ...
user avatar
0 votes
0 answers
3 views

I do not understand normalization and standardization

in my ML Course at Uni, a big topic was normalization and standardization, but still - I don't really get why we do that. More specifically, I'm working on a (fairly complex) CNN to predict missing ...
user avatar
1 vote
0 answers
17 views

What methods I could use to analyze the contingency table?

I am data science beginner, and I have a question about methods that I could use to analyze the following data. It is a simple case, I am trying to check the influence of cohabitation before marriage ...
user avatar
  • 11
0 votes
0 answers
20 views

How to Calculate values inside DBLP collaboration un-directed network? | Using python

(EDITED- thanks) I have loaded txt file (com-dblp.ungraph.txt) which includes network to a COLAB notebook. I want to calculate describe vals of the net in the COLAB- how can I do it? Number of nodes ...
user avatar
0 votes
0 answers
10 views

Fit Regression Model: Variables Dependent by Interval

I want to use the "Fit Regression Model" tool in Minitab with a series of predictors and one response. I am trying to reduce the number of variables I use as predictors. There are three ...
user avatar
0 votes
0 answers
7 views

Estimating effect of coefficient in log-linear model

I'm working on a log-linear model which has multiple variables with their coefficients. The model estimates a variable log(Y) ~ X. I'm trying to estimate the effect ...
user avatar
0 votes
0 answers
10 views

Capacity planning and modelling

I have a business case in which I am going to model how many devices are required given the predicted workload in a series of monthly cohorts in the next ten years. The work could come from multiple ...
user avatar
  • 763
0 votes
0 answers
17 views

Kullback Leibler divergence of two datasets having different sizes

is there a way to apply KL on 2 datasets of different sizes? ps: my datasets are 2 heat maps with 120x180 and 60x90 as width and height respectively.
user avatar
  • 23
0 votes
0 answers
16 views

Anomaly Detection in a time series data using statistical methods

0: I've been working on time series data recently and need to detect the anomalies observed using HoltWinters(Triple exponential smoothing) model. Base: I implemented Triple exponential smoothing, ...
user avatar
0 votes
0 answers
7 views

Split new values based on historical data - best methods

I have daily hitorical data, something like this (Mon - 100, Tue - 100, Wed - 70, Thu - 110, Fri - 200, Sat - 100, Sunday - 30). I have several values for the same historical period. Next I have a new ...
user avatar
0 votes
0 answers
13 views

for which Machine learning models we should Discrete continuous features?

my question may be seems duplicate but its because that I couldn't find any clear and unequivocal answer for this question on the web. 1. I want to know that when should I discrete and categorize ...
user avatar
  • 3
0 votes
0 answers
12 views

How do I compare value counts in each year when each year has a different sample size?

Let's say I went out each year asking people whether they really liked Pepsi, did not care about it, or hated it. But in the first year I asked people at my company when it was small, then the second ...
user avatar
0 votes
0 answers
9 views

Metrics for changes in datapoints

I am looking for a metric which quantifies the noise in data with respect to neighbouring data points. I have time series data and I want to check how much values change from a previous to a following ...
user avatar
0 votes
0 answers
22 views

Is it a right way to score the importance of categorical features against categorical targets using ANOVA?

I know that f_classif method is mostly used for scoring numerical features and categorical targets. But I'm wondering if it's possible to score features coded by integers or binary codes against ...
user avatar
  • 101
2 votes
1 answer
31 views

How to generate a rule-based system based on binary data?

I have a dataset where each row is a sample and each column is a binary variable. The meaning of $X_{i, j} = 1$ is that we've seen feature $j$ for sample $i$. $X_{i, j} = 0$ means that we haven't seen ...
user avatar
0 votes
0 answers
9 views

why zero centring of data from activation function is good for deep nural network?

I was reading an article that mentioned reasons why tanh is better than sigmoid and one reason was that tanh gives zero-centered data but I couldn't understand why and how it will affect our network. ...
user avatar
0 votes
0 answers
19 views

uncertainties in non-convex optimization problems (neural networks)

How do you treat statistical uncertainties coming from non-convex optimization problems? More specifically, suppose you have a neural network. It is well known that the loss is not convex; the ...
user avatar
  • 3
0 votes
0 answers
17 views

Polynomial regression with two variables. How can I find expressions to describe the coefficients?

I'm not sure if this is an appropriate place for this question, so please feel free to redirect me if it is not. I just moved it from Super User, where it seemed like there weren't many similar ...
user avatar
  • 101
0 votes
0 answers
10 views

Recommendations for modelling panel data

sending positive wishes to y'all. I have about 10 years of growth rates in real estate prices and some other macroeconomic variables such as inflation, unemployment rates, fuel prices, growth in ...
user avatar
0 votes
1 answer
38 views

What are the requirements for a word list to be used for Bayesian inference?

Intro I need an input file of 5 letter English words to train my Bayesian model to infer the stochastic dependency between each position. For instance, is the probability of a letter at the position 5 ...
user avatar
  • 3
0 votes
0 answers
5 views

How would I check the validity of covariates in my linear model on several hundred datasets?

I have this linear model with predictors that I need to prove are statistically significant and pass the necessary lm assumptions. I know for a single dataset, I can use various LM tests, but the ...
user avatar
0 votes
1 answer
17 views

Interpreting interaction term coefficient in GLM/regression

I'm a psychology student and trying come up with a research plan involving GLM. I'm thinking about adding an interaction term in the analysis but I'm unsure about the interpretation of it. To make ...
user avatar
0 votes
1 answer
23 views

what degree of fredom one should use while calculating standard diviation for standardizing data

I am writing a function to standardize the data and I found out that we can choose either ddof = 0 or ddof = 1, so I got confused that which one to choose and why? Does this make any difference?
user avatar
0 votes
0 answers
27 views

Feature importance of a linear regression

What is the easiest and easy to explain feature importance calculation for linear regression? I know I can use Shap to compute feature importance, but I find it difficult to explain it to stakeholders,...
user avatar
  • 209
0 votes
0 answers
15 views

Can we use an independent t-test as a metric for feature importance?

I have a supervised binary classification problem. I tuned an xgboost model on the training set and achieved a reasonably high accuracy on the test set. Now I want to interpret the results of the ...
user avatar
  • 165
0 votes
0 answers
11 views

List of main statistics models

I am not able to find some list of main statistics models. Is is possible to devide statistics models into categories as supervised (regression,classification) x unsupervised (clustering) or is it ...
user avatar
0 votes
0 answers
4 views

Fisher's exact test when resampling from the population

I am using different ML classifiers for making predictions related to a binary classification. I would like to compare two groups. Let's call them A and B. The already trained and tested classifier ...
user avatar
0 votes
0 answers
15 views

How to do binning in matrix data

I have some data like ...
user avatar
0 votes
0 answers
19 views

How to interpret arg min in the the following equation?

I am studying the following equation: $\hat{s}_m(n) = \text{arg}\text{min}_{s_m(n)\in A_s}|\frac{\psi_m^H}{||\psi_m^H||^2}y_m(n)-s_m(n)|^2$----(1) here $A_s$ is 1x$N$ vector of QPSK symbols, $s_m(n)$ ...
user avatar
  • 1
1 vote
1 answer
46 views

Incorrect example of applying Bayes theorem

I have been reading the book "The Data Science Design Manual" (by Steven S. Skiena) and I came across an example that explained how the Bayes theorem can be applied that confused me and made ...
user avatar
  • 111
0 votes
1 answer
13 views

How to find an anomalous matrix among many?

Let's say we have a bunch of matrices that we know are non-anomalous. We now receive a new matrix and want to know if it belongs into the group or is way off. Is there a way to do that? I'm thinking ...
user avatar
0 votes
0 answers
34 views

Is there a Solution Manual for Introduction to Statistical Pattern Recognition Book by Keinosuke Fukunaga?

I'm looking for a solution manual for Introduction to Statistical Pattern Recognition Book by Keinosuke Fukunaga ? I couldn't find anything on the internet . If you have any solution manual by ...
user avatar
0 votes
0 answers
9 views

Statistically increase mean difference between two data sets?

A dataset will be used to train a binary classification model. For better understanding/visualization, the data set was divided into 2: one set with all the rows that result in prediction value of 1 ...
user avatar
  • 1
1 vote
1 answer
43 views

What's the purpose of statistical analysis ( statistically important features) vs feature elimination in machine learning

I am developing a classification model for covid19 symptoms (after being ill) and I don't understand statistical analysis importance (some parts of it) 1 Firstly: Basically we perform statystical ...
user avatar
  • 113
0 votes
0 answers
33 views

Comparing performances of different models using hypothesis testing

A common workflow in applied ML is you train different models on some data and evaluate it on a particular test set. I have often seen people just selecting a ML metric based on their requirements and ...
user avatar
  • 63
0 votes
0 answers
10 views

what is meant by walk square in a maximum likelihood estimate

I have some data about maximum likelihood estimates. We have fields of estimate, standard error, pr>chiSq and Walk Square. I have no idea what the Walk Square is referring to - it feels like some ...
user avatar
  • 1
1 vote
1 answer
11 views

How should I write the conclusion when doing the same hypothesis on two different subjects and get different results?

I am doing a research trying to find if there's a correlation between twitter sentiments and sales and I'm doing it on two different companies. They are in the same industry and direct competitors. ...
user avatar
0 votes
0 answers
24 views

How to sample a dataframe or numpy array with a particular interval?

I have the following dataframe : ...
user avatar
  • 1
0 votes
0 answers
13 views

Analyzing series of events

noob in data science here. I would like some advice on the following: I have a series of events that can happen at random times of a day, there is the event start/end time. If an event started already ...
user avatar
0 votes
0 answers
9 views

Maximum entropy optimization for approximating image quality score distribution - as in Google's "Neural Image Assessment" paper

I am asking this question after a thorough research on the internet and having read every single detail of "NIMA: Neural Image Assessment" by Hossein Talebi and Peyman Milanfar. Before ...
user avatar
1 vote
0 answers
11 views

P-value using Gaussian Discriminate Analysis

I was wondering , in Gaussian Discriminate Analysis (GDA) model, say we have two classes to classify y=0 and y=1 So after fitting the Gaussian over y=0 and y=1 dataset, when we try to predict class ...
user avatar
0 votes
0 answers
10 views

Filling NaN values

According to my knowledge, before filling nan values we have to check whether data is missing because of MCAR, MAR or MNAR and it depends on how features are correlated with each other and then make a ...
user avatar
0 votes
0 answers
24 views

Is this a successful implementation of KL Divergence from scratch, and how can I graph all distributions?

I'm attempting to implement a KL Divergence on two imaginary dice from scratch and I'm not sure if it is correct. By 'how do I graph all distributions' I'm wondering if the KL Divergence just a number ...
user avatar
  • 157
0 votes
0 answers
6 views

Model transfer with limit to none label information

I have this problem I hope to get some help here. Say I have a type of product A whose measurements are X_A and an outcome property is ...
user avatar
  • 21
1 vote
0 answers
35 views

Using F_regression to find the best significant features

We are trying to use SelectKBest F_Regression scoring function on a pool of 1000 numerical features, and solve a regression problem. Also, we wanted to paralellize the execution of SelectKBest and we ...
user avatar
0 votes
0 answers
10 views

How to test if a curve is well described by an ellipse?

I have a set of data points in 2D, and I am trying to come up with some sort of statistical to determine if the points fall along an ellipse. My idea so far is to fit an ellipse to the points, take ...
user avatar
  • 1
0 votes
0 answers
26 views

When linear regression is really a good model to be used?

By definition of linear regression model: If we note $Y$ the real random variable to be explained (endogenous variable, dependent or response) and $X$ the explanatory variable or fixed effect (...
user avatar
1 vote
1 answer
17 views

Finding contribution/weight of features in output from a series of data

I have a dataset which consists of the following features : a1,a2,.....aN.... sum of all the features together gives a constant output say 100. For eample the dataset may look like this. a1 a2 ...
user avatar
  • 11

1
2 3 4 5
21