# Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

961 questions
Filter by
Sorted by
Tagged with
7 views

### Causal Inference where the treatment assignment is randomized

I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on ...
5 views

### Minimal N days to reject sharpe ratio

A strategy assumes to have an annual sharp ratio of 8. Now backtest on N days trading data. If the N days return is negative, how many days does N need to reject the assumption that the annual sharp ...
6 views

### Two-sided grubbs test KeyError: 355

I am trying to this two sided grubbs test by passing in a pandas.Series object and an appropriate alpha value. whenever I do the test on the whole dataset, I have ...
6 views

### Sample covariance between a quantitative variable and a binary encoded variable

I have the following question: What does the sample covariance between a quantitative variable and a binary encoded variable tells us? And would it make sense to include the binary variable in a ...
12 views

### Why do I get this result with a chi square test?

I have a question about the chi square independence test, I'm working on dataset and I'm interested in by the link between the categorie of product and the gender, I plot my contingency table. ...
7 views

### Lower bound of confidence interval of a test data

I am interested in studying the effect of increasing data samples for a regression model on train error and test error. For this I have used confidence intervals for different values of a sample data. ...
13 views

### Diebold-Mariano test for comparing multivariate models

I want to know if exist a Diebold-Mariano test version for comparing the accuracy of predictions between multivariate statistics models. For example, compare a VAR model and multivariate GARCH ...
28 views

### How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
12 views

### Weighted Voting: Accuracy and Coverage in Class Weight

I have a data set in which the data is coming from various sources. Approx 3k records were verified manually and respective source is tagged if the data comes from that source and is valid/correct. I ...
28 views

### A data in which an employee id is given in multiple months and many categorical features are there. To predict future retention. Recommend what to do?

I have this dataset in which we have to predict the retention of employees,i.e. how much will an employee stay in a company? This seems easy but the main obstruction here is that the same employee_id ...
9 views

### how to convert an array of non regularly interleaved coordinates to a matrix of weights using interpolation to obtain uniform sampling

I have an array of coordinates each one with an associated timestamp. Something like: ...
15 views

### Is there any library to perform robust clustering given two probability distribution with noise?

Given a dataset $X$ consisted with $w|X|$ samples drawn from a mixture of multivariate Gaussian distributions (say in two dimensions) and $(1-w)|X|$ samples of noise, is there any ...
8 views

### Is it okay to use high dimensional data for QQ plots comparison?

I had a question regarding using high-dimensional data for a QQ plot. I have some audio signal data with dimensions: data.shape = (3035, 180) I first used this code:...
27 views

### Why is pearson correlation popular if it detects only linear correlation?

Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
17 views

### Right way to compare model scores for Next Best Action

I have around 15 classification models for different products built in different ways (some are RF, some are Gradient Boosting, some were downsampled in one way, others in other way, some are built in ...
48 views

### Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
14 views

### Checking Skweness for each column in Dataset

Is checking Skweness for each column before feeding it to the algorithm a mandatory step while preprocessing our dataset or on what conditions does checking skewness depends? Currently I am working on ...
25 views

### Distribution Shift vs Transfer Learning

Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem  ...
13 views

### How to implement the contribution analysis using PCA?

I have been looking into implementing the Q-Residual and Hotelling's T statistics calculation to the PCA components which is similar to the following article and website: Structural Health Monitoring ...
19 views

### How to determine elastic net models coefficient significance?

I have a small dataset with just 160 data points. When I tried ordinary linear regression on the data, I could not add more than four features without vif inflating greater than 5 (I made a stepwise ...
29 views

### Residual plot understanding

I am trying to build a regression model to predict Gerrit code review delay (i.e the time between the creation time of the code review until the time of the last update.) For that, I used a random ...
51 views

### Why is it important that a transformation applied to a metric is monotonic?

What does it mean for a function to be monotonic? Why is it important that a transformation applied to a metric is monotonic?
26 views

### Can dummy variables be treated as continuous in statistical models? [closed]

Can dummy variables be treated as continuous in statistical models?
90 views

### Is my approach about the ML model correct?

First of all, I am a newbie here and it is my first question on this platform, so I apologize for the mistakes about the format if there are any. In my thesis study, I am trying to identify the non-...
37 views

### How is a time series with strong seasonality and trend stationary?

I was playing around with time series and stationarity and stumbled on something which I'm not getting completely. There are two popular ways of testing out stationarity - ADF and KPSS. And I was ...
14 views

### cbind doesn't attribute the name of the column

I'm trying to create a new data frame that contains the row names and the first column of an existing data : i tried this ...
34 views

### Can I compare two models trained on different but similar datasets to help find differences between the two datasets?

I have a multivariate dataset the contains A and B. I want to see if there are differences between the A and B samples. I currently have two ideas on how to do this, but I am not sure if they are ...
21 views

### Does time-series dependence penalize prediction with CNNs?

I'm trying to use CNNs on time series data (EEG), measured on different people. Each person has 10-20 recorded signals of different lengths and every subject has one global class assigned. Example: ...
29 views

### Confidence intervals for evaluation on test set

I'm wondering what the "best practise" approach is for finding confidence intervals when evaluation the performance of a classifier on the test set. As far as I can see, there are two ...
34 views

### How to approach this: Percentage change in one KPI leading to change in other KPIs?

I want to know how can I approach or model this problem. I have 7 KPIs (3 of them dependent on each other) and one main KPI (total 8 KPIs). I want to understand effect of these 7 KPIs on the main KPIs....
32 views

### log(odds) to p formulation

$$Log(Odds) = log({p \over (1-p)})$$ $${p \over (1-p)} = e^{b+b_1x_1+....}$$ I understand up to here, however how does this: $$p = (1-p) e^{b+b_1x_1+...}$$ become:  p = {1 \over {1+e^{-(b+b_1x_1+......
10 views

### Urgent help on STATA! Panel regression, regressing lagged version of variable with itself in particular industry across few industries! [closed]

Just some kind help on the syntax for the above... Basically this is an event analysis whereby there are 10 sectors, and for instance quarterly data, so therefore, mergers that took place in a ...
38 views

### Calculate the standard error of the difference between two independent proportions

Edit: This may be better for stats stack exchange but I'm in a data mining class and we use R so I'll ask it here as well just in case anyone knows how to do this with R instead of manually. Here's ...
50 views

### Is the explained variance a good metric for autoencoders?

I want to evaluate how an autoencoder will perform on my data. Now, I can do this with the mean squared error of the decoded data compared to the original data, and this is fine when comparing this ...
37 views

### Why scikit-learn's sequential feature selection requires how much features to be selected beforehand?

From the version 0.24, the scikit-learn has new method 'SequentialFeatureSelector', which adds (forward selection) or removes (backward selection) features to form a feature subset in a greedy fashion....
14 views

### How to create transition probability (state) for q-learning algorithm designed to control traffic light system using python?

I am trying to create a q learning algorithm to control traffic light systems. I am representing the state with a matrix : ...
6 views

### Understanding forward process in diffusion models

I was reading a blog on diffusion models where I came across this expression. I didn't understand why it is \begin{align} \sqrt[]{1-\beta \small{t}}*\large{x}\small{t-1} \end{align} and what exactly ...
139 views

### Where can I find study materials? [closed]

Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics? Thanks
5 views

### Obtaining multiple DAGs from Observational Data

I wish to obtain the most probable set of DAGs created from observational data and then compare them using my own score function. Ideally, I would want to give the observational data and then the ...
15 views

### How to perform a Monte Carlo simulation with continuous sampling using discrete quantiles?

Assume I have registered the duration of 10 tasks and built the table below with using this data: Duration For how many tasks it happened 4 days 5 task 6 days 2 task 8 days 2 task 10 days 1 task ...
26 views

### I am not sure if I used the right hypotheses test or made the right conclusions, could you give me a feedback?

I have just finish a basic statistic course and I decided to apply what I learned in a small project using the wine quality dataset. I am not sure if what I did was correct, so if possible, I would ...
10 views

### Creating a popularity index from multivariate data

I have some data from an ecommerce website with features like product_name, product_category product_link, product_id, free_delivery(1 or 0), price, discount, avg_rating, number of reviews, ...
10 views

### Why GP posterior variance is the worst-case error?(exact proof)

I am reading this paper, which explains the connecting idea Gaussian Process and Kernel methods in detail. I am impressed by the insightful explanation in this paper, but am stuck on one part in ...
20 views

### What is the difference between a hypothese teste made with a normal distribution and a one sample t-test? And When to use each one?

I am studying statistics on my own and I am not understanding when to use a one sample t test or a test with a normal distribution. As I understand, both are comparing a population mean with a sample ...
132 views

### Correlation between continuous variables and multi class categorical variables in python

I was trying to figure out a way of finding a correlation between continuous variables and a non-binary target categorical label. The only thing I though of is by fitting the labels into Multinomial ...
19 views

### Mutiple binary classification for for best propensity to buy one of the product

Problem:- I have 5 products for sell and I can pitch only one product in a month to one customer.so I wants to know which product customer can buy. Proposed solution:- I build 5 binary logistic models ...
9 views

### Bayesian interference - interpretation of results of two groups

I'm trying to understand Bayesian interference on example the comparison of two groups from this source. It's not all too clear about interpretation of some parameters in summary, like: mcse_mean ...
12 views

### How to find the lag that shows highest correlation between two time series variables?

I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...