Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Does severe multicollinearity affect solving linear regression by gradient descent?

Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario? Does it converge at the minima? (My guess is, Cost function of linear ...
1
vote
0answers
10 views

Spread of infection: how to correct for population size

The Covid-19 death data for different countries should reflect how fast the infection spreads through the population and how susceptible the population is. The total number of deaths after some longer ...
1
vote
1answer
13 views

SPC vs Autoencoders in anomaly detection

Considering the usage of Autoencoders in anomaly detection of time-series data, why SPCs (control charts) have lost their charm? Are there any advantages with Autoencoders and disadvantages with SPCs?
1
vote
1answer
8 views

What is the best approach for send time optimization? [closed]

I could no find a lot information about how the companies doabout send time optimization, either for push notifications or email campaigs. having historical data about clicks and sends what would be ...
0
votes
0answers
7 views

When I use bootstrapping to calculate confidence intervals for accuracy, should I resample the entire dataset or just the test set?

In this article, the author resamples the entire dataset and retrains the model each time. However, I don't see why you couldn't just resample the test set only, which is what I'm currently doing. In ...
1
vote
1answer
17 views

Are month and weekday rulers of surveys? How can I test the validity?

I am analyzing a survey and came across an interesting pattern. Weekday and month seems to have a very high say in the score given by customers. There can be many reasons for this, though it is ...
1
vote
0answers
16 views

Find the mode value and frequency in R

I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...
1
vote
0answers
10 views

How effective is Moore Penrose for solving regression problems with overdetermined system of equations?

For regression problems with #Predictors > #observations, I recently read about Moore Penrose (pseudo inverse method) which solves the problem of non invertible matrix in OLS for regression problems. ...
1
vote
1answer
12 views

How to identify recurring patterns in this set of transactional data

I'm working on a dataset of banking transactions and would like to find recurrent transactions. I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa....
0
votes
1answer
19 views

How to compute Hessian matrix for log-likelihood function for Logistic Regression

I am currently studying the Elements of Statistical Learning book. The following equation is in page 120. It calculates the Hessian matrix for the log-likelihood function as follows But is the ...
-1
votes
0answers
10 views

how to generate variates from a particular categorical binary data?

So im working with more than 100 thousand samples dota2 dataset which consist of the winner and the "hero" composition from each match. I was trying to build winner of the match prediction model ...
0
votes
1answer
14 views

How to perform link prediction in text based relationship data

I need to establish if there is a link between 2 columns from two different datasets with one matching column, where; ...
0
votes
0answers
11 views

Is there a statistical law that can determine quantity of experiment and predict accuracy of results in my experimental context? [migrated]

I designed a biological protocol A which measures a continuous variable qA for each sample I designed a biological protocol B which measures a continuous variable qB for each same sample as above ...
0
votes
0answers
7 views

How to find average lag time with variance & confidence of two variables

I have two variables as time series, one a consequent of the other, I would like to find the average time delay it takes the dependent variable to act on the independent variable. Additionally, I ...
0
votes
1answer
14 views

Using a past time series to predict how a present time series will pan out?

Let's say that I have past data indicating how some time series panned out. Now I also have the beginnings of a new time series that I expect to pan out in a similar trend to the old one. What are ...
4
votes
2answers
107 views

Determine statistically whether new product cannibalise old product by using data

Assume that $A$ is a cab company which offers online cab booking through their standard account. Recently, the company launched a pre-paid premium account with features such as discounted rides,...
6
votes
2answers
79 views

How do I test a difference between two proportions representing fatality rate for Covid 19 in Philippines and World (except Philippines)?

I'm trying to analyse if the fatality rate from my country (A third world country) vary significantly from the world's fatality rate. So I'd basically have two samples, labeled (Philippines) and (...
0
votes
0answers
16 views

How much can I reduce sample size in a bootstrap used for density estimation?

I have a dataset with 90m rows by 10 numeric/boolean columns. I need to calculate bootstrap distributions for 40 statistics. These statistics are definitely not Gaussian (nor similar to the mean of ...
3
votes
1answer
99 views

Does ridge regression always reduce coefficients by equal proportions?

Below is an excerpt from the book Introduction to statistical learning in R, (chapter-linear model selection and regularization) "In ridge regression, each least squares coefficient estimate is ...
2
votes
0answers
14 views

overlapping of surfaces with holes

I have two disks, of equal diameter. Both disks have a 'spatter' of holes, in random positions (from a two-dimensional - i.e., surface - uniform distribution; in other words, holes close to the sides ...
0
votes
1answer
39 views

(Basic) statistics

I got a misunderstanding regarding basic statistics (I think) but I can't get my head around that: I did an online survey regarding the usage of some application. The user could answer 1 (I don't know ...
0
votes
0answers
21 views

Bias of an estimator?

What is p(X;0)actually modelling? What's the difference between the two types of theta? What's happening in this equation and did expectation convert into summation ? Source:https://subscription....
1
vote
2answers
29 views

How can I find to starting point of skewed data in python?

I have a list like this, ...
1
vote
1answer
29 views

How to adaptively sample n-dimensional data and build an optimum training set

My input space is at least 10 dimensional (after reducing it by various component analysis such as PCA) and the output space is 4 dimensional. I am building a neural network that works something like ...
4
votes
1answer
25 views

Does k fold cross validation become less useful when number of observations is very large?

As seen in the accepted answer for variance of k-fold cross validation , the simulation shows that k-fold CV has the same test error rate for different values of k when n=200. Does this mean that k-...
0
votes
0answers
25 views

What's the way to categorize user if user can fall into multiple group

Let's assume I have a sample data looks like this ...
3
votes
2answers
81 views

Correlation vs Coefficient of multiple linear regression

I have 10000 samples. There are 4 independent variables and 1 dependent variable. The independent variables are all centered with 0 mean. I found the correlation between each of these variables ...
0
votes
3answers
30 views

weather and products sell relationship which one should be the target variable?

Let say I have this dataset kor_cat is the categories of the product like(noodle,cookie....) and the kor_qty is the number of product selled I want to know the weather has a relation with the ...
0
votes
0answers
17 views

statistical hypothesis and rejection

Recently I have to analyze the multiclass classification problem But, it's not just I have to predict and submit I have to make a hypothesis with this data and have to find the rejection methods for ...
1
vote
0answers
21 views

MinMaxScaling vs L1/L2-Normalization

I'm wondering about the difference or the application of the different types of rescaling data. So far, I'm aware that standardization assumes the data has a gaussian distribution. So if this is the ...
0
votes
0answers
31 views

Generalised Estimating Equation (GEE) vs. Recurrent Neural Network (RNN)

Has anyone looked into or know what is the difference between a GEE model and an RNN model in terms of what these two models are doing? Apart from the differences in structure of these two models ...
0
votes
0answers
19 views

Variable selection involving mixture of numerical, high cardinal,low cardinal features

Consider a dummy dataframe: A B C D …. Z 1 2 as we 2 2 4 qq rr 5 4 5 tz rc 9 This dataframe has 25 independent variables and one target variable ,the ...
6
votes
3answers
373 views

Are Undergraduate Statistics Concepts Used in Practice? [closed]

I'm curious for more experienced Data Scientist, have you ever used t - test, ANOVA, Wilcoxon, etc? Basically my question is, do you perform inference task, or purely prediction tasks? (Machine ...
0
votes
1answer
27 views

Why i am getting t statistics different

import scipy.stats as st st.t.ppf(0.95, df=9) Output: 1.8331129326536335 As per this, I should get 2.262 with dof = 9.
3
votes
1answer
22 views

Can numerical discrete finite data be always treated also as categorical?

In many sources, for example here data is classified as being qualitative (categorical) and quantitative (numerical). Where numerical data can be continuous or discrete, and discrete can be finite or ...
1
vote
1answer
25 views

Require Guidance - Identifying risk factors

I have a dataset of patients who have been inpatients (admitted to hospital) and not admitted (but visited as outpatients). Class proportion is 66:34. I have collected a list of features for all ...
0
votes
1answer
25 views

How to interpret axis of Histogram and distribution curve?

After doing statistical analysis of data I got the below figures: The data contains a line length in meters which is represented with Line(m). The purpose here is to confirm the distribution behavior ...
2
votes
0answers
17 views

How can I test the relationship between events and categorical data

I have a dataset comprised of mostly categorical data, in particular, different name tags of events that happen within a process. Those are accompanied of their <...
2
votes
0answers
47 views

What is a logworth statistic, and how useful is it?

My teacher mentioned it today, and there is nearly zero good search results for it, other than one mention each in the SAS and JMP documentation. It says it is ...
1
vote
2answers
76 views

Anomaly Detection/Novelty detection

I have a data-set that has over 6 million normal data and around 50 anomaly data.Those anomaly data is identified by manually(monitoring the user`s activity over camera and identify). I need to ...
1
vote
0answers
17 views

how to predict the time series data with two target outputs

I am a newbie to time series problems, I am doing one of the time series problems, I a have 317 columns and, I need to predict the stock Revenue and Price, I am trying to do it, but the many columns ...
1
vote
1answer
19 views

Rules/Methods to use for determining user groups with no review data

I’m working on an analytical problem where I have customers who deal with end consumers. End consumers can leave customers reviews based on their experience (Star ratings, text etc). I’m trying to ...
3
votes
0answers
28 views

Approaches to choosing number of bins in histogram

Choosing the number of bins in a histogram has always been something that gets me thinking a lot. Based on the number of bins chosen, the graph at time looks a lot different and also could lead to ...
0
votes
0answers
21 views

Scoring products across all stores

I am trying to score each product based off a couple of features. I was thinking I can use the sales rate(7 days prior) as the target column and predict the sales rate using some ensemble technique ...
0
votes
1answer
23 views

Quantifying a trade-off

Let our data be a set of 2d points sampled from a line $ax+by=1$ (that we do not know) plus some noise $\nu$. This is meant to represent a frontier of development - we can build a car with a ...
0
votes
0answers
29 views

How to perform TTest on multiple coliumns

My dataframe is below ...
3
votes
1answer
55 views

Confidence value for face recognition

In the context of face recognition I have the following histogram: blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the ...
1
vote
1answer
21 views

Knowing Joint probability distribution between feature-label space

I am doing a course CORNELL CS4780 "Machine Learning for Intelligent Systems". you can find the link here for the one I am going to refer 1st lecture The professor explains, we have a sample $D ={...
3
votes
1answer
51 views

Risk prediction vs classification model

I am working on a binary classification model. Currently, when I use scikit logistic regression, it outputs binary values like 0s and 1s. However, I understand, from online reading, that it outputs ...
1
vote
0answers
13 views

On ttest why t-stat is greater than one?

I have my data in two culomns as below: On excel did this ttest and this is my result: I can not Understand why Hypothesized Mean Difference =0 but tStat = -1.6367...? I mean my NULL/Zero ...

1
2 3 4 5
14