Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
3
votes
1answer
24 views

Confidence value for face recognition

In the context of face recognition I have the following histogram: blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the ...
1
vote
1answer
13 views

Knowing Joint probability distribution between feature-label space

I am doing a course CORNELL CS4780 "Machine Learning for Intelligent Systems". you can find the link here for the one I am going to refer 1st lecture The professor explains, we have a sample $D ={...
2
votes
1answer
23 views

Risk prediction vs classification model

I am working on a binary classification model. Currently when I use Scikit logistic regression, it outputs me binary values like 0s and 1s. But I understand through online reading that it outputs ...
1
vote
0answers
12 views

On ttest why t-stat is greater than one?

I have my data in two culomns as below: On excel did this ttest and this is my result: I can not Understand why Hypothesized Mean Difference =0 but tStat = -1.6367...? I mean my NULL/Zero ...
2
votes
0answers
50 views

Multidimensional collaborative filtering model

I have a dataset that is approximately structured in the following way: 500 users, 500 products, 100 countries, 2 seasons, 300000 ratings. Meaning that I have 300,000 rows containing unique ...
2
votes
0answers
18 views

Difference between Gibbs sampling and variational Bayes inference

After reading in blogs and books, I came to the conclusion that Gibbs sampling and variation Bayes are methods for estimating or inference of posterior. Below link described but it's difficult to ...
0
votes
0answers
9 views

Fleiss' Kappa when different raters rate different subsets of examples?

The Wikipedia article on Fleiss' Kappa cites Fleiss as stating that Fleiss' Kappa does not require all raters to have rated all examples. However, the equation given on Wikipedia as the definition of $...
2
votes
1answer
29 views

Encoding correlation

I have rather theory-based question as I'm not that experienced in encoders, embeddings etc. Scientifically I'm mostly oriented around novel evolutionary model-based methods. Let's assume we have ...
2
votes
0answers
38 views

Probability of the recipient to open the email

I am trying to build a model using logistic regression, where my dependent variable is y=1 if the mail was opened, y=0 if it was not. I have data approximately 10 records (10 rows) for every ...
2
votes
1answer
25 views

Compare between similar and dissimilar couples of instances

I label couples of similar and dissimilar instances based on user behavior. each instance has a lot of features. I have few ways of labeling the couples. I know want to evaluate which of the label ...
5
votes
1answer
214 views

Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67. I had the same algorithms and same set of hyperparamters during yesterday and ...
1
vote
0answers
15 views

Statistical test using Gmean to compare multiple algorithms on multiple datasets

I am new in this area. I am facing some issues while comparing the algorithms using statistical test. I have following result of Gmean of some classification algorithm. Abalone, Balance-scale, Car, ...
0
votes
1answer
26 views

Does variance and standard deviation both measure how spread out the numbers are?

I've heard from different sources that, standard deviation measures how spread out the numbers are. But I've also heard the same for variance. Is it technically correct to say this statement for both ...
0
votes
1answer
52 views

What is the difference between Freidman mse and mse?

The original question was posted on StackOverflow. However, taking into account the recommendations of Sergey Bushmanov, I'm providing an answer through this medium.
1
vote
0answers
25 views

Seeking advice: How would you define “top performer” with a set of data?

still consider myself a noobie in ds here, so I may be asking a very easy and stupid question, but would really appreciate some guidance. The hypothetical situation is that, you have a website where ...
1
vote
1answer
27 views

How to use scikit metrics for a statsmodel or vice versa?

Am working on binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3558. I did refer this post but not sure whether it is updated now or anyone has any way to compute this ...
2
votes
1answer
43 views

How to Bootstrap dataset for 10000 AUC scores?

I am new to ML and trying to learn the nuances. I work on a binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3554. What I currently do is 1) split the data into train(...
2
votes
1answer
26 views

Why and how to match variables in logistic regression?

I have a dataset of ~4.7K records focused on binary classification with 60 features. class 1 is of 1554 records and class 2 is of 3558 records. Now I would like to find the risk factors that ...
1
vote
1answer
32 views

Threshold to consider new feature as a new finding to a model?

I am working on binary classification problem with 5K records and 60 features. Through feature selection, I narrowed it down to 14 features. In existing literature, I see that there are well-known 5 ...
5
votes
1answer
637 views

Is there any difference between a weak learner and a weak classifier?

I am reading about Gradient Boosting, AdaBoost etc. I have found the following two concepts weak learner and weak classifier. Are they the same? If there is any difference what is it?
1
vote
1answer
20 views

How is a coincidence matrix constructed for computing Krippendorff's alpha?

I am looking at two documents to help me learn about constructing coincidence matrices in order to gain a better understanding of Krippendorff's alpha. I am using these two: https://repository.upenn....
2
votes
2answers
32 views

Which metric to choose for tracking model performance?

I am working on a binary classification problem with class proportion of 33:67. Currently what I am doing is running multiple models like LR,...
0
votes
0answers
54 views

Why a significant risk factor doesn't increase AUC-score/F1-metric?

I have a binary classification problem with 5K records and 60+ features/columns/variables. dataset is slighlt imbalanced with 33:67 class proportion What I did was 1st) Run a logistic regression (...
1
vote
1answer
28 views

How to interpret statsmodel output - logit?

I ran a logit model using statsmodel api available in Python. I have few questions on how to make sense of these 1) What's the difference between ...
2
votes
1answer
19 views

evaluation metrics for multiple values per session

I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based ...
1
vote
1answer
49 views

difference between feature interactions and confounding variables

Let me define the problem space. I am working a binary classification problem. I am trying to build a causal model as well as predictive model. My aim is to find list of significant features (based ...
2
votes
1answer
48 views

Why is T test reweighting on a word X word co-occurrence matrix so effective?

I am going through Stanford NLP class: http://web.stanford.edu/class/cs224u/ A task in the homework is to implement T-test reweighting on a word X word co-occurrence matrix: https://nbviewer.jupyter....
8
votes
2answers
89 views

Which statistical test tells which classifier performs better than the other?

I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do ...
3
votes
2answers
25 views

Statistical learning for data-limited systems

I'm currently conducting a review for quantitative methods being used for tropical inland fisheries. One of the major problems for modeling methods in tropical inland fisheries is the lack of data ...
0
votes
0answers
11 views

What is the likelihood of this function

I have been given that the outputs from a feed-forward network ( one output ) follows$ Y |X = x, w ∼ N(\hat{y} (x, w), σ^2)$ * From what i know, the likelihood function is defined as $p_{X|\theta}...
0
votes
1answer
42 views

Trouble in calculating the covariance matrix

I'm trying to calculate the covariance matrix for a dummy dataset using the following formula, but it's not matching with the actual result. Let's say the dummy dataset contains three features, #...
0
votes
0answers
11 views

Bayesian/Frequentist “philosophies” in relation to common ML models

Starting to think about questions tout could come up in a data science job interview, there is this one question that is bugging me since i don't have a deep background in statistics. I have a ...
0
votes
0answers
8 views

How to summarize/describe feature trends with fix number of scalars?

I'm trying to make feature extraction from features with numbers over different timespans (like stock values movement) and I want to encode these trends into a fix number of descriptive scalars. I ...
0
votes
0answers
14 views

Bradley-Terry for pairwise comparison with confidence levels

For a scientific study that I'm currently working on, I will evaluate three different configurations (A - B - C). These configurations will be evaluated in a pairwise comparison fashion (A-B, A-C, B-C,...
0
votes
1answer
31 views

Evaluating likelihood of egg breaking when falling in random container on concrete [closed]

I am working on a project where I would like to predict whether an egg will break if it is put in a container that is then dropped on concrete. I am looking at the different factors that play a role ...
0
votes
0answers
12 views

Structural Risk Minimization: Inequality in “Foundations of Machine Learning”

I am reading the chapter about structural risk minimization in Mohri's book. However, I have trouble understanding the derivation of inequality 4.5 on page 83. More precisely, why is $$\sum_k \...
0
votes
1answer
11 views

Understanding AC_errorRate loss function

I'm reading an article about Rolling Window Regression: a Simple Approach for Time Series Next value Predictions. He explains about 5 different loss functions. I managed to understand the first four, ...
2
votes
1answer
22 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
2
votes
1answer
36 views

Diff. in P-value & F-Stat. Multiple linear regression

Even if we have individual p-values for each predictor. Why do we need overall F-statistic? I read this solution but I am not sure if I get it right. Can someone please explain? Source: "An ...
0
votes
1answer
24 views

Customer Intelligence - How to measure success?

we are creating models that aim to filter new leads from our current customer base. We started to create propensity models that calculate a percentage for each customer for a certain product group. I ...
1
vote
1answer
32 views

Is there a common strategy to measure if a difference-significance of two areas under two ROC curves

I conduct sound detection experiments with mice. I have a stimulus sound and a "noise" sound that shoukd be ignored. I want to measure how well the mouse ignors the noise (with respect to, say, ...
4
votes
1answer
39 views

Naive Bayes implementation: why Laplace smoothing is different from theory?

Let's have a Naive Bayes Bernoulli classifier with $n_C$ classes and $n_F$ features. According to the formula in here and here and almost every theory book I could see, Laplacian smoothing means that ...
0
votes
0answers
9 views

What is an appropriate statistical method for determining correlations between variables with likert scale data?

I am conducting an exploratory analysis with a dataset of about 200 records (all likert scales (numeric)). I aim to determine the correlations between the different variables that the responses ...
1
vote
0answers
12 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
1
vote
0answers
21 views

Hypothesis Testing- Independent variable Importance

I am learning Data Science and I have a confusion in one topic. I would like to describe my approach. I have understood the problem statement. I have used snowflake schema to join tables that have ...
0
votes
1answer
25 views

How does $\chi^2$ feature selection work?

I can't find the information how $\chi^2$ are used to select numerical features for a model. ...
2
votes
0answers
13 views

Error when drawing random numbers from a custom continuous distribution using scipy.rv_continuous

I am trying to generate a sample of random numbers from a custom distribution $$ p(x) = x^{n}e^{-xtn}. $$ After reading the tutorial on scipy's website, I wrote a subclass which I called kbayes: <...
0
votes
0answers
10 views

Wavelet transform to identify 'meaningful' data changes

Can I use the wavelet transform to find the 'meaningful' changes in a series of 1d data? For example I have a series of integers representing angles of line segments. The majority of these values are ...
0
votes
0answers
70 views

Statistical loss function for categorical distributions

For training an autoencoder model whose outputs (and inputs) are parameters from a categorical distribution $[q_1, q_2, \ldots, q_n]$, I have to define a proper loss function measuring the distance ...
0
votes
1answer
37 views

confusion on outliers

I am not able to distinguish the outliers - When to go with std. dev and When do we need to go with Median. My understanding on std. dev. is - if the data is away from mean by more than 2 std dev. we ...