Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

How to find equilibrium distribution based on a timeseries of its driver

I'm investigating how the distribution of a variable (a quality measure) varies over the last couple of years (quality measure on x-axis, lower is better): We know that these distributions are ...
0
votes
0answers
8 views

What statistical test should I use (SPSS)?

So this is probably asking a bit much but I'm not sure what test I should use: I'm planning on investigating the significance of socioeconomic status when considering the health effects of workplace ...
0
votes
1answer
45 views

When R2 score and MSE are not correlated

I'm training some forecasting models and then, to check performance I see several metrics. It's surprising for me when they are no related, for example: Let's suppose I'd have two models, A and B. --&...
0
votes
0answers
12 views

Backing into a statistical significance level for a proportion

I'm doing some natural language processing which involves assigning labels to a sample of news articles. Its multilabel classification so an article might receive label1, label2, both, or neither. I'...
1
vote
0answers
18 views

With a 5000x20 CSV as data input discover the most common occurrences of numbers in a row

As input I have a CSV with 5000 lines (and growing) and 20 fixed columns containing a number from 1-80. A row may look like this. Is it possible using Orange3 to analyze each row and find out what ...
1
vote
1answer
13 views

T-test against normalised or standardised data gives different results

I am studying the problem to predict popularity of a tweet, and want to test null hypothesis: there is no relationships between favorite_counts and another set of variables, like number of friends of ...
0
votes
0answers
11 views

How to do ? serious skewed right transformations normally distribution

My target is doing two-way Anova. But Here is some problem. This is my data like I try log(X). Looks like a normal distribution. But try Kolmogorov-Smirnov prove is not. Can anyone recommend another ...
1
vote
0answers
13 views

How to perform and interpret Redundancy Analysis (RDA)?

I generated a data.frame (database) where I have the arimetic average with water quality parameters, land use data and the associated monitoring stations over time. I would like to know if the ...
0
votes
1answer
24 views

Recommended number of features for regression problem

In the following link the answer recommends a feauture amount of N/3 for regression (or it is quoted). Where N corresponds to the sample size: How many features to sample using Random Forests Is there ...
0
votes
0answers
16 views

How contextual bandit use single model out put q-value for each arm?

I've been studying Thompson Sampling and one of the bottleneck is that when recommending 100 items to users, each item has beta distribution thus need to select ...
1
vote
0answers
13 views

Handling imbalanced Feature (X) not lavbel (Y) in machine learning

I am very new to this field and have done a decent amount of research on this, but every time, I stumble upon handling the imbalanced label by using f1 score, recall, precision as metrics, and using ...
1
vote
2answers
43 views

For a student who is a beginner in quantitative research and statistics, which is the better statistical tool to start: R or IBM SPSS? Why? [closed]

Currently, I am writing my research design. However, I am still indecisive on what statistical tool should I use for the data analysis. I tried looking up on the internet and there are disparate ...
1
vote
0answers
10 views

Fail to decompose and make stationary time series

I am looking for some suggestions for my time series. I am dealing with the column "Temperature (C)" from this dataset. I am trying to make it stationary in order to do some forecasting on ...
0
votes
1answer
35 views

Two-parametric transformation of Box-Cox vs Yeo–Johnson transformation

I choose which transformation to use for my data (data contains both positive and negative values). Wikipedia says the following: The Yeo – Johnson transformation allows also for zero and negative ...
1
vote
1answer
14 views

How to evaluate goodness of fit of a sinusoidal model using an F-Test?

I have 2000 samples of unevenly sampled timeseries data collected at 2 hour intervals over the course of 24 hours. Working in Python, I used scipy.optimize.curve_fit...
0
votes
1answer
22 views

Can I leave natural outliers in a dataset in training?

Can I leave unedited natural outliers in a dataset (outliers that have not appeared just because of mistyping of mistakes in the data)? Or should I also remove them or change them?
0
votes
0answers
9 views

Subtraction of two variances (scores)

I was wondering, would it be correct to say, when we treat two variances of two populations as a random variable itself (or as a score), that we can simply get a resultant variance V_subtract = V_pop1 ...
0
votes
1answer
16 views

Should I consider unreadable records in confusion matrix while calculating accuracy?

I have 6 classes in my dataset and model. Dataset is regarding ECG signal Having x number of records for each of these classes. The confusion matrix looks like this - My question is, should I ...
0
votes
1answer
13 views

Calculating statistical ranks between datasets with unpaired observations

The problem is the following: I have multiple datasets for which I want to calculate a ranking for each. All observations contained in the datasets can be arbitrarily permuted, so they are unpaired, ...
1
vote
1answer
21 views

Persistence and stationarity together

I am trying to analyse a time series. I want to get only quantitative results (so, I'm excluding things like "looking at this plot we can note..." or "as you can see in the chart ...&...
0
votes
0answers
11 views

Calculate rate from related datasets

I have the monthly sales rate for various products. The products are sold in different countries. I'm looking for a meaningful way to calculate the sales rate at each country. The sales rate indicated ...
1
vote
1answer
13 views

The fine line dividing ML modelling and statistical modelling

I've been thinking about the difference between ML modelling and statistical modelling. I would to ask, on a philosophical level, is my thinking correct: modelling is basically a process of fitting a ...
3
votes
2answers
61 views

Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
1
vote
1answer
28 views

How to analyze repeated measure data for prediction?

In my work, we collect sales data of our products. We have a set of 1st level customers (lets call that group as jacks) with whom we do we business. These jacks then sell our products to end customers ...
1
vote
0answers
17 views

How should I think when I want to compare mu and sigma for different images in VAE?

I'm searching for a way to compare mu and sigma values of the encoder network's output of variational autoencoders. In detail, imagine I trained my VAE on the MNIST digits dataset using the official ...
0
votes
0answers
16 views

Automatize autocorrelation in python

I'm trying to automatize my autocorrelation study in Python. My question is: is it possible? Let me explain. I have a time series and I just learnt how to interpret the autocorrelation plot. My ...
1
vote
0answers
15 views

Causal Inference where the treatment assignment is randomized [closed]

I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on ...
0
votes
0answers
8 views

Minimal N days to reject sharpe ratio

A strategy assumes to have an annual sharp ratio of 8. Now backtest on N days trading data. If the N days return is negative, how many days does N need to reject the assumption that the annual sharp ...
0
votes
0answers
11 views

Two-sided grubbs test KeyError: 355

I am trying to this two sided grubbs test by passing in a pandas.Series object and an appropriate alpha value. whenever I do the test on the whole dataset, I have ...
1
vote
0answers
8 views

Sample covariance between a quantitative variable and a binary encoded variable

I have the following question: What does the sample covariance between a quantitative variable and a binary encoded variable tells us? And would it make sense to include the binary variable in a ...
0
votes
0answers
13 views

Why do I get this result with a chi square test?

I have a question about the chi square independence test, I'm working on dataset and I'm interested in by the link between the categorie of product and the gender, I plot my contingency table. ...
0
votes
0answers
7 views

Lower bound of confidence interval of a test data

I am interested in studying the effect of increasing data samples for a regression model on train error and test error. For this I have used confidence intervals for different values of a sample data. ...
0
votes
0answers
22 views

Diebold-Mariano test for comparing multivariate models

I want to know if exist a Diebold-Mariano test version for comparing the accuracy of predictions between multivariate statistics models. For example, compare a VAR model and multivariate GARCH ...
1
vote
1answer
28 views

How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
0
votes
0answers
14 views

Weighted Voting: Accuracy and Coverage in Class Weight

I have a data set in which the data is coming from various sources. Approx 3k records were verified manually and respective source is tagged if the data comes from that source and is valid/correct. I ...
0
votes
0answers
29 views

A data in which an employee id is given in multiple months and many categorical features are there. To predict future retention. Recommend what to do?

I have this dataset in which we have to predict the retention of employees,i.e. how much will an employee stay in a company? This seems easy but the main obstruction here is that the same employee_id ...
0
votes
0answers
9 views

how to convert an array of non regularly interleaved coordinates to a matrix of weights using interpolation to obtain uniform sampling

I have an array of coordinates each one with an associated timestamp. Something like: ...
1
vote
0answers
16 views

Is there any library to perform robust clustering given two probability distribution with noise?

Given a dataset $X$ consisted with $w|X|$ samples drawn from a mixture of multivariate Gaussian distributions (say in two dimensions) and $(1-w)|X|$ samples of noise, is there any ...
1
vote
0answers
13 views

Is it okay to use high dimensional data for QQ plots comparison?

I had a question regarding using high-dimensional data for a QQ plot. I have some audio signal data with dimensions: data.shape = (3035, 180) I first used this code:...
1
vote
1answer
28 views

Why is pearson correlation popular if it detects only linear correlation?

Pearson's correlation coefficient is widely used to check for relationship between predictors in a dataset. However, since it measures only linear relationships between variables, wouldn't it be ...
1
vote
0answers
18 views

Right way to compare model scores for Next Best Action

I have around 15 classification models for different products built in different ways (some are RF, some are Gradient Boosting, some were downsampled in one way, others in other way, some are built in ...
0
votes
1answer
59 views

Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
0
votes
0answers
14 views

Checking Skweness for each column in Dataset

Is checking Skweness for each column before feeding it to the algorithm a mandatory step while preprocessing our dataset or on what conditions does checking skewness depends? Currently I am working on ...
1
vote
1answer
27 views

Distribution Shift vs Transfer Learning

Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem [1] ...
0
votes
0answers
13 views

How to implement the contribution analysis using PCA?

I have been looking into implementing the Q-Residual and Hotelling's T statistics calculation to the PCA components which is similar to the following article and website: Structural Health Monitoring ...
0
votes
0answers
19 views

How to determine elastic net models coefficient significance?

I have a small dataset with just 160 data points. When I tried ordinary linear regression on the data, I could not add more than four features without vif inflating greater than 5 (I made a stepwise ...
0
votes
0answers
29 views

Residual plot understanding

I am trying to build a regression model to predict Gerrit code review delay (i.e the time between the creation time of the code review until the time of the last update.) For that, I used a random ...
1
vote
0answers
61 views

Why is it important that a transformation applied to a metric is monotonic?

What does it mean for a function to be monotonic? Why is it important that a transformation applied to a metric is monotonic?
2
votes
1answer
90 views

Is my approach about the ML model correct?

First of all, I am a newbie here and it is my first question on this platform, so I apologize for the mistakes about the format if there are any. In my thesis study, I am trying to identify the non-...
0
votes
0answers
37 views

How is a time series with strong seasonality and trend stationary?

I was playing around with time series and stationarity and stumbled on something which I'm not getting completely. There are two popular ways of testing out stationarity - ADF and KPSS. And I was ...

1
2 3 4 5
20