Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
1answer
12 views

How do i iterate functions until convergence in R?

I am looking to iterate until convergence but I am not sure how i should code it. A simple similar example is something like $X_0=5 , Y_0=3$ for $n=1,2..$ $P_n=X_{n-1}-Y_{n-1}$ $Y_n=3P_n$ $X_n=P_n+...
1
vote
0answers
6 views

Interpreting statsmodel Granger Causality test results

I am trying to use statsmodels Granger Causality test: https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.grangercausalitytests.html to assess whether "positivity score" ...
1
vote
1answer
18 views

How to perform a running (moving) standardization for feature scaling of a growing dataset?

Let's say that there is a function $r$ $r_n = r(\tau_n)$, where $n$ denotes a so-called time-step of a system with an evolving state. Both $\rho$ and $\tau$ should equally influence $r$, and should ...
0
votes
0answers
14 views

Using deep learning or statistical tools to complete physical equation

I need to predict a continuous variable Y in a regression setting. Theory in the field tells me it should follow an equation of the form: $$Y=g_0(X)-f_1(X)g_1(X)-f_2(X)g_2(X)$$ Where X is a vector of ...
1
vote
1answer
33 views

Computing adjusted p-values in batches

Data For simplicity I propose the following scheme: I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous: ...
0
votes
0answers
30 views

What metrics to use in regression if variance in output label is very low?

What metrics/method should one adopt in judging the error when variance is low in output variable. To give you an example : the output variable can be stock prices over a month, the variance generally ...
0
votes
0answers
8 views

Weighted Average Score

I have several different scores that I'd like to combine them into a single score that represents all parameters. Score components are: Nutrition Score (1-10): 1 for the worst quality and 10 for the ...
1
vote
2answers
20 views

Modality of data

Can anyone please explain in clear words what is generally meant by "modality of data"? I know what modality means with respect to distributions.
1
vote
3answers
539 views

What is this type of plot and how to interpret it?

I was presented to this plot and i've seen it before, however i would like to know what does the countour around the main line mean. Ps: i couldn't find enough information about it to investigate by ...
1
vote
1answer
24 views

Permutation test on two groups

I am trying to use a permutation test to test my hypothesis. I want to make sure I am understanding concept of permutation correctly. I have control and experimental group. Then I combine them and ...
0
votes
1answer
11 views

How to calculate similarity between 2 users based on the images they share?

Say there are 2 users, A and B, and they each shared 10 images (in some social media site), which I have collected in 2 folders separately. I want to calculate the similarity between the 2 users based ...
0
votes
1answer
23 views

What to choose for null hypothesis and alternate hypothesis?

In this article which explains types of t-tests for “Independent Two-Sample t-test” example: We can confirm that the t-statistic is again less than the t-critical value so we fail to reject the null ...
0
votes
2answers
18 views

How to calculate steady/incremental growth?

I have timeseries data for stocks at minute intervals. What is the best way to calculate incremental growth, for example if I have a stock's price from 9am to 2pm at minute intervals, how can calcule ...
0
votes
0answers
18 views

How to develop a likelihood based prediction model to predict chance of rain in a particular hour of a year

I have time-series weather data (from 2005-2018) of temperature and rainfall (every one-hour interval) from three different location weather stations. I want to predict/see what likely be a chance of ...
4
votes
2answers
93 views

Stacking and Ensembling methods in Data Science

I understand that using stacking and ensembling has become popular, and these methods can give better results than using a single algorithm. My question is: What are the reasons, statistical or ...
1
vote
1answer
16 views

Why do we add “αd” to N in Laplace Smoothing?

I just started to learn Naive Bayes algorithm. Then I learned to use Laplace smoothing to avoid getting probability of zero. I understand the purpose of using it, but, in the expression of Laplace ...
24
votes
7answers
6k views

Is Python a viable language to do statistical analysis in?

I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
1
vote
0answers
23 views

Adjusting Variables in Multiple Linear Regression

Suppose I have $10$ exposure variables $a_{1}...a_{10}$ and one dependent outcome $y$.I suspected $a_{9}$ and $a_{10}$ as a possible confounding variables. So at first I performed the multiple linear ...
1
vote
1answer
29 views

Random Sampling

If I have a bag filled with numbered balls. The numbers are from say 1 to 10. But the number of balls follow poisson distribution with lambda=1. Just a graph like below Now I want to make sample of ...
1
vote
1answer
27 views

Is rolling of biased dice random phenomenon?

Random phenomenon is a situation in which we dont know what the out come is going to come. Rolling of unbiased dice is a random phenomenon since we dont know what number is going to come. We can only ...
0
votes
0answers
18 views

Off-Policy Estimation - Importance Sampling with Negative Rewards

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
-1
votes
0answers
15 views

Odds ratios explanation and question

I am very new to the odds ratios and I am trying to learn by myself and by watching videos and reading articles and trying to figure out a solution for one of my statistics questions. So I have read ...
-1
votes
1answer
18 views

What are some theoretical approches to study Machine Learning/Deep Learning?

I've recently studied "basics of machine learning" (chapter 5) in Deep Learning by Ian Goodfellow. They're explaining ML with most of statistical and probabilitical methods. But I wish to ...
0
votes
2answers
33 views

How can I explain this chart showing 5-days moving average?

I have plotted the frequency of items sold through time, trying to determine the trends by moving average. I considered a 5-days window. I would like to know if this approach makes sense and how I ...
1
vote
2answers
24 views

how can i compare two classification algorithms?

all, i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. ...
-1
votes
1answer
39 views

deep learning and uncertainty estimation

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML ...
0
votes
1answer
19 views

Use both differencing and normalization in time series modeling to make it stationary?

I am working on a time series dataset. Should we use both differencing and normalizing or either of the ones to make it stationary?
1
vote
0answers
35 views

Trusting p-values when errors are not normally distributed [duplicate]

Suppose I have a multiple linear regression model and the errors are not distributed normally.Does the central limit theorem hold true in this case? should i trust the p values of the coefficients of ...
4
votes
2answers
212 views

a data point - too little to spend resources during data preparation?

I am learning analytics online and have some quick questions. Usually when we do analysis, why is that we usually ignore the items/data points that are less frequent? Let's say for ex: we have is ...
0
votes
0answers
6 views

monte carlo simulation for impact to revenue by adjusting credit limits

I've been tasked with prioritizing projects for the year. One of them is to estimate the impact to net revenue if we were to adjust the purchase limits for our users. Our platform is transactional in ...
1
vote
1answer
21 views

Using BM25 to rank words

How effective is it to use BM25 to rank words, to be more specific i have a dictionary of words and i want to rank only words in a document that are also in my dictionary. I want to rank all words in ...
5
votes
2answers
93 views

Why is 10 considered the default value for k-fold cross-validation?

I understand very well what k-fold cross-validation is. In my studies, and at work, I've always heard something along the lines of: We most often use k=10 ...
0
votes
0answers
19 views

How to convert google trend relative values into more “absolute” values?

Searching key words or terms on google trends provides a time series of relative search numbers. I want to be able to analyse these trends more accurately compared to general trends in the same ...
3
votes
2answers
65 views

Distribution vs Histogram [closed]

I am not very good at understanding statistics jargon I tried to read articles on statistics every article says distribution and show a picture of bell curve. I understand that x axis of bell curve ...
1
vote
0answers
13 views

testing statistical significance when comparing regression models

I have 3 models, a random forest, a XGBoost and a baseline regression model. I've performed a 5-fold cross validation on all models and use MAE as the scoring metric. So this means I basically have 3 ...
0
votes
0answers
15 views

Sample size calculation for AB testing

I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My ...
0
votes
0answers
11 views

Treatment and Control selection in A/B Testing

I'm hoping to get a better understanding of A/B Testing design. In particular, I'm interested in understanding how treatment and control units are selected. I read that these 2 groups are selected ...
0
votes
1answer
22 views

How to adjust/smooth a certain number using constants or rules

Hi, I am handling a dataset with a customer purchase history. The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage. There are two ...
0
votes
0answers
8 views

Non-monotone missing data, and inverse probability weighting

I'm having difficulty identifying whether or not my missing data pattern is 'monotone'. I have two variables with missing data, and the missing data patterns in each variable do not completely ...
1
vote
1answer
23 views

Linear regression: variables with high P-value

Often, after training a linear regression model on data, some variables/features would have high P-value, which means they are not statistically significant. Although there are automated methods like ...
0
votes
1answer
46 views

Relationship between two continuous variables in time series data

I have a dataset that collects daily data based on transactions between two entities. I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of ...
2
votes
2answers
19 views

Notation for features (general notation for continuous and discrete random variables)

I'm looking for the right notation for features from different types. Let us say that my samples as $m$ features that can be modeled with $X_1,...,X_m$. The features Don't share the same distribution (...
0
votes
0answers
30 views

How to estimate tail distribution for computing Conditional Value at Risk in python?

How to estimate parameteres of generalized pareto distribution in python? 🙏 I solve the task of computing Value at Risk and Conditional Value at Risk. ...
6
votes
2answers
192 views

How distribution of data effects model performance?

I am working on House Prices: Advanced Regression Techniques dataset. I was going through some kernels noticed many people converted SalePrice to ...
0
votes
1answer
13 views

pass variable length argument to mstats.kruskalwallis

I am trying to run kruskawallis test on multiple columns of my data for that i wrote an function ...
4
votes
2answers
149 views

Mean estimation for nested location data

I want to estimate the average income for a location. I have nested data in the following way: A block is inside a neighborhood, which is inside a zipcode, which is inside a district, which is inside ...
3
votes
1answer
51 views

Replace human judgements with mathematical approach/theories

I would like to give a context of what I did. 1) Let's say there are two dictionaries (dict A and dict B) each containing a list of words/terms as shown below. 2) Now my task is to find matching ...
2
votes
1answer
16 views

Topology of Data

Do every data resembles some form of space like manifold? Meaning, do geometry of data (especially the big data) is a manifold? Is every data embedded to some sphere? If that is so, can we also say ...

1
2 3 4 5
16