Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Causal Inference where the treatment assignment is randomized

I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on ...
0
votes
0answers
5 views

Minimal N days to reject sharpe ratio

A strategy assumes to have an annual sharp ratio of 8. Now backtest on N days trading data. If the N days return is negative, how many days does N need to reject the assumption that the annual sharp ...
0
votes
0answers
6 views

Two-sided grubbs test KeyError: 355

I am trying to this two sided grubbs test by passing in a pandas.Series object and an appropriate alpha value. whenever I do the test on the whole dataset, I have ...
1
vote
0answers
6 views

Sample covariance between a quantitative variable and a binary encoded variable

I have the following question: What does the sample covariance between a quantitative variable and a binary encoded variable tells us? And would it make sense to include the binary variable in a ...
0
votes
0answers
12 views

Why do I get this result with a chi square test?

I have a question about the chi square independence test, I'm working on dataset and I'm interested in by the link between the categorie of product and the gender, I plot my contingency table. ...
0
votes
0answers
7 views

Lower bound of confidence interval of a test data

I am interested in studying the effect of increasing data samples for a regression model on train error and test error. For this I have used confidence intervals for different values of a sample data. ...
0
votes
0answers
13 views

Diebold-Mariano test for comparing multivariate models

I want to know if exist a Diebold-Mariano test version for comparing the accuracy of predictions between multivariate statistics models. For example, compare a VAR model and multivariate GARCH ...
1
vote
1answer
28 views

How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
0
votes
0answers
12 views

Weighted Voting: Accuracy and Coverage in Class Weight

I have a data set in which the data is coming from various sources. Approx 3k records were verified manually and respective source is tagged if the data comes from that source and is valid/correct. I ...
0
votes
0answers
28 views

A data in which an employee id is given in multiple months and many categorical features are there. To predict future retention. Recommend what to do?

I have this dataset in which we have to predict the retention of employees,i.e. how much will an employee stay in a company? This seems easy but the main obstruction here is that the same employee_id ...
0
votes
0answers
9 views

how to convert an array of non regularly interleaved coordinates to a matrix of weights using interpolation to obtain uniform sampling

I have an array of coordinates each one with an associated timestamp. Something like: ...
1
vote
0answers
15 views

Is there any library to perform robust clustering given two probability distribution with noise?

Given a dataset $X$ consisted with $w|X|$ samples drawn from a mixture of multivariate Gaussian distributions (say in two dimensions) and $(1-w)|X|$ samples of noise, is there any ...
1
vote
0answers
8 views

Is it okay to use high dimensional data for QQ plots comparison?

I had a question regarding using high-dimensional data for a QQ plot. I have some audio signal data with dimensions: data.shape = (3035, 180) I first used this code:...
1
vote
1answer
27 views

Why is pearson correlation popular if it detects only linear correlation?

Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
1
vote
0answers
17 views

Right way to compare model scores for Next Best Action

I have around 15 classification models for different products built in different ways (some are RF, some are Gradient Boosting, some were downsampled in one way, others in other way, some are built in ...
0
votes
1answer
48 views

Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
0
votes
0answers
14 views

Checking Skweness for each column in Dataset

Is checking Skweness for each column before feeding it to the algorithm a mandatory step while preprocessing our dataset or on what conditions does checking skewness depends? Currently I am working on ...
1
vote
1answer
25 views

Distribution Shift vs Transfer Learning

Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem [1] ...
0
votes
0answers
13 views

How to implement the contribution analysis using PCA?

I have been looking into implementing the Q-Residual and Hotelling's T statistics calculation to the PCA components which is similar to the following article and website: Structural Health Monitoring ...
0
votes
0answers
19 views

How to determine elastic net models coefficient significance?

I have a small dataset with just 160 data points. When I tried ordinary linear regression on the data, I could not add more than four features without vif inflating greater than 5 (I made a stepwise ...
0
votes
0answers
29 views

Residual plot understanding

I am trying to build a regression model to predict Gerrit code review delay (i.e the time between the creation time of the code review until the time of the last update.) For that, I used a random ...
2
votes
0answers
51 views

Why is it important that a transformation applied to a metric is monotonic?

What does it mean for a function to be monotonic? Why is it important that a transformation applied to a metric is monotonic?
1
vote
0answers
26 views

Can dummy variables be treated as continuous in statistical models? [closed]

Can dummy variables be treated as continuous in statistical models?
2
votes
1answer
90 views

Is my approach about the ML model correct?

First of all, I am a newbie here and it is my first question on this platform, so I apologize for the mistakes about the format if there are any. In my thesis study, I am trying to identify the non-...
0
votes
0answers
37 views

How is a time series with strong seasonality and trend stationary?

I was playing around with time series and stationarity and stumbled on something which I'm not getting completely. There are two popular ways of testing out stationarity - ADF and KPSS. And I was ...
2
votes
1answer
14 views

cbind doesn't attribute the name of the column

I'm trying to create a new data frame that contains the row names and the first column of an existing data : i tried this ...
2
votes
1answer
34 views

Can I compare two models trained on different but similar datasets to help find differences between the two datasets?

I have a multivariate dataset the contains A and B. I want to see if there are differences between the A and B samples. I currently have two ideas on how to do this, but I am not sure if they are ...
1
vote
0answers
21 views

Does time-series dependence penalize prediction with CNNs?

I'm trying to use CNNs on time series data (EEG), measured on different people. Each person has 10-20 recorded signals of different lengths and every subject has one global class assigned. Example: ...
1
vote
1answer
29 views

Confidence intervals for evaluation on test set

I'm wondering what the "best practise" approach is for finding confidence intervals when evaluation the performance of a classifier on the test set. As far as I can see, there are two ...
1
vote
1answer
34 views

How to approach this: Percentage change in one KPI leading to change in other KPIs?

I want to know how can I approach or model this problem. I have 7 KPIs (3 of them dependent on each other) and one main KPI (total 8 KPIs). I want to understand effect of these 7 KPIs on the main KPIs....
1
vote
2answers
32 views

log(odds) to p formulation

$$Log(Odds) = log({p \over (1-p)}) $$ $${p \over (1-p)} = e^{b+b_1x_1+....}$$ I understand up to here, however how does this: $$p = (1-p) e^{b+b_1x_1+...}$$ become: $$ p = {1 \over {1+e^{-(b+b_1x_1+......
1
vote
0answers
10 views

Urgent help on STATA! Panel regression, regressing lagged version of variable with itself in particular industry across few industries! [closed]

Just some kind help on the syntax for the above... Basically this is an event analysis whereby there are 10 sectors, and for instance quarterly data, so therefore, mergers that took place in a ...
3
votes
1answer
38 views

Calculate the standard error of the difference between two independent proportions

Edit: This may be better for stats stack exchange but I'm in a data mining class and we use R so I'll ask it here as well just in case anyone knows how to do this with R instead of manually. Here's ...
2
votes
1answer
50 views

Is the explained variance a good metric for autoencoders?

I want to evaluate how an autoencoder will perform on my data. Now, I can do this with the mean squared error of the decoded data compared to the original data, and this is fine when comparing this ...
0
votes
1answer
37 views

Why scikit-learn's sequential feature selection requires how much features to be selected beforehand?

From the version 0.24, the scikit-learn has new method 'SequentialFeatureSelector', which adds (forward selection) or removes (backward selection) features to form a feature subset in a greedy fashion....
1
vote
0answers
14 views

How to create transition probability (state) for q-learning algorithm designed to control traffic light system using python?

I am trying to create a q learning algorithm to control traffic light systems. I am representing the state with a matrix : ...
0
votes
0answers
6 views

Understanding forward process in diffusion models

I was reading a blog on diffusion models where I came across this expression. I didn't understand why it is \begin{align} \sqrt[]{1-\beta \small{t}}*\large{x}\small{t-1} \end{align} and what exactly ...
1
vote
2answers
139 views

Where can I find study materials? [closed]

Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics? Thanks
0
votes
0answers
5 views

Obtaining multiple DAGs from Observational Data

I wish to obtain the most probable set of DAGs created from observational data and then compare them using my own score function. Ideally, I would want to give the observational data and then the ...
0
votes
0answers
15 views

How to perform a Monte Carlo simulation with continuous sampling using discrete quantiles?

Assume I have registered the duration of 10 tasks and built the table below with using this data: Duration For how many tasks it happened 4 days 5 task 6 days 2 task 8 days 2 task 10 days 1 task ...
1
vote
0answers
26 views

I am not sure if I used the right hypotheses test or made the right conclusions, could you give me a feedback?

I have just finish a basic statistic course and I decided to apply what I learned in a small project using the wine quality dataset. I am not sure if what I did was correct, so if possible, I would ...
0
votes
0answers
10 views

Creating a popularity index from multivariate data

I have some data from an ecommerce website with features like product_name, product_category product_link, product_id, free_delivery(1 or 0), price, discount, avg_rating, number of reviews, ...
2
votes
0answers
10 views

Why GP posterior variance is the worst-case error?(exact proof)

I am reading this paper, which explains the connecting idea Gaussian Process and Kernel methods in detail. I am impressed by the insightful explanation in this paper, but am stuck on one part in ...
1
vote
1answer
20 views

What is the difference between a hypothese teste made with a normal distribution and a one sample t-test? And When to use each one?

I am studying statistics on my own and I am not understanding when to use a one sample t test or a test with a normal distribution. As I understand, both are comparing a population mean with a sample ...
0
votes
0answers
132 views

Correlation between continuous variables and multi class categorical variables in python

I was trying to figure out a way of finding a correlation between continuous variables and a non-binary target categorical label. The only thing I though of is by fitting the labels into Multinomial ...
0
votes
0answers
19 views

Mutiple binary classification for for best propensity to buy one of the product

Problem:- I have 5 products for sell and I can pitch only one product in a month to one customer.so I wants to know which product customer can buy. Proposed solution:- I build 5 binary logistic models ...
0
votes
0answers
9 views

Bayesian interference - interpretation of results of two groups

I'm trying to understand Bayesian interference on example the comparison of two groups from this source. It's not all too clear about interpretation of some parameters in summary, like: mcse_mean ...
0
votes
0answers
12 views

How to find the lag that shows highest correlation between two time series variables?

I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
0
votes
1answer
29 views

How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?

We want to introduce a new price list for the customers of our international SaaS company. Beforehand we want to test this new price list in several countries. A/B test cannot be conducted here ...
0
votes
0answers
10 views

Hypothesis testing using permutation test method, how to reverse of null hypothesis?

My goal is to prove two groups A,B comes from the same distribution therefore I could use A as control and B as experimental for ABtest. KPI is set to CTR(#clicks/#impressions). Therefore need to ...

1
2 3 4 5
20