Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
1
vote
1answer
14 views

How can I compare the grammatical complexity between two texts using their sentences dependency length?

This is a continuation to the following thread. I have two texts, common English texts such as news articles and informative texts versus a technical textbook. I want to compare the grammatical ...
1
vote
0answers
12 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
1
vote
0answers
25 views

Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
0
votes
0answers
38 views

How to use historic data (granularity at day level) for ML modeling?

There is a scenario where I have to use historic data which is at the day level for the past 5 years.Actually it is water flow data, what quantity of water was flown on that particular day. I have to ...
1
vote
1answer
39 views

What conclusion can I get when the variable is influenced by other but there isn't any correlation?

I am doing an analytic exploratory analysis. If the target is a continuous variable and the attributes are all categorical (discrete values), in order to know if exist any influence on the target from ...
0
votes
3answers
45 views

How to convert regression into classification?

So I have a regression problem with bunch of features X, and labels in the amount (price $). How can I convert it to classification problem? I have read about convert label from continuous to ...
0
votes
0answers
38 views

Do I need to use learning algorithms for this type of data?

I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or ...
-3
votes
1answer
61 views

What is the difference between statistical-analysis and meta-analysis? [closed]

Meta analysis has been developed to integrate findings of studies.Is it part of statistical analysis? Alternatively, does it have a unique foundation.
1
vote
1answer
22 views

How to avoid confusion between Percentage and Percentage Points?

Whenever I have to communicate a change in a numerical metric, I found myself struggling if I should use Percentage or Percentage Points. Especially, when the metric is a percentage itself, it can be ...
1
vote
1answer
16 views

Advise resources on un-supervised learning [closed]

I have seen that people coming into data science will rush into scikit-learn or other libraries without trying to learn the knowledge behind. Its good to follow a top-down approach but most of times ...
0
votes
2answers
23 views

how to use predictions on a single value?

I am comfortable using Machine learning on my train data and test data and validate it. But the question here is if I want to predict a single variable how do I do it? Let's suppose I have done ...
0
votes
1answer
27 views

How do i iterate functions until convergence in R?

I am looking to iterate until convergence but I am not sure how i should code it. A simple similar example is something like $X_0=5 , Y_0=3$ for $n=1,2..$ $P_n=X_{n-1}-Y_{n-1}$ $Y_n=3P_n$ $X_n=P_n+...
2
votes
1answer
42 views

Interpreting the results based on Granger Causality test

I am trying to use Granger Causality test: https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.grangercausalitytests.html to assess whether "positivity score" affects ...
2
votes
1answer
22 views

How to perform a running (moving) standardization for feature scaling of a growing dataset?

Let's say that there is a function $r$ $r_n = r(\tau_n)$, where $n$ denotes a so-called time-step of a system with an evolving state. Both $\rho$ and $\tau$ should equally influence $r$, and should ...
0
votes
0answers
16 views

Using deep learning or statistical tools to complete physical equation

I need to predict a continuous variable Y in a regression setting. Theory in the field tells me it should follow an equation of the form: $$Y=g_0(X)-f_1(X)g_1(X)-f_2(X)g_2(X)$$ Where X is a vector of ...
2
votes
1answer
45 views

Computing adjusted p-values in batches

Data For simplicity I propose the following scheme: I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous: ...
0
votes
0answers
31 views

What metrics to use in regression if variance in output label is very low?

What metrics/method should one adopt in judging the error when variance is low in output variable. To give you an example : the output variable can be stock prices over a month, the variance generally ...
1
vote
0answers
9 views

Weighted Average Score

I have several different scores that I'd like to combine them into a single score that represents all parameters. Score components are: Nutrition Score (1-10): 1 for the worst quality and 10 for the ...
1
vote
2answers
27 views

Modality of data

Can anyone please explain in clear words what is generally meant by "modality of data"? I know what modality means with respect to distributions.
3
votes
3answers
557 views

What is this type of plot and how to interpret it?

I was presented to this plot and i've seen it before, however i would like to know what does the countour around the main line mean. Ps: i couldn't find enough information about it to investigate by ...
2
votes
1answer
28 views

Permutation test on two groups

I am trying to use a permutation test to test my hypothesis. I want to make sure I am understanding concept of permutation correctly. I have control and experimental group. Then I combine them and ...
0
votes
1answer
18 views

How to calculate similarity between 2 users based on the images they share?

Say there are 2 users, A and B, and they each shared 10 images (in some social media site), which I have collected in 2 folders separately. I want to calculate the similarity between the 2 users based ...
0
votes
1answer
23 views

What to choose for null hypothesis and alternate hypothesis?

In this article which explains types of t-tests for “Independent Two-Sample t-test” example: We can confirm that the t-statistic is again less than the t-critical value so we fail to reject the null ...
0
votes
2answers
19 views

How to calculate steady/incremental growth?

I have timeseries data for stocks at minute intervals. What is the best way to calculate incremental growth, for example if I have a stock's price from 9am to 2pm at minute intervals, how can calcule ...
5
votes
2answers
104 views

Stacking and Ensembling methods in Data Science

I understand that using stacking and ensembling has become popular, and these methods can give better results than using a single algorithm. My question is: What are the reasons, statistical or ...
1
vote
1answer
18 views

Why do we add “αd” to N in Laplace Smoothing?

I just started to learn Naive Bayes algorithm. Then I learned to use Laplace smoothing to avoid getting probability of zero. I understand the purpose of using it, but, in the expression of Laplace ...
25
votes
7answers
6k views

Is Python a viable language to do statistical analysis in?

I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
1
vote
0answers
26 views

Adjusting Variables in Multiple Linear Regression

Suppose I have $10$ exposure variables $a_{1}...a_{10}$ and one dependent outcome $y$.I suspected $a_{9}$ and $a_{10}$ as a possible confounding variables. So at first I performed the multiple linear ...
1
vote
1answer
29 views

Random Sampling

If I have a bag filled with numbered balls. The numbers are from say 1 to 10. But the number of balls follow poisson distribution with lambda=1. Just a graph like below Now I want to make sample of ...
1
vote
1answer
27 views

Is rolling of biased dice random phenomenon?

Random phenomenon is a situation in which we dont know what the out come is going to come. Rolling of unbiased dice is a random phenomenon since we dont know what number is going to come. We can only ...
0
votes
0answers
20 views

Off-Policy Estimation - Importance Sampling with Negative Rewards

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
-1
votes
1answer
19 views

What are some theoretical approches to study Machine Learning/Deep Learning?

I've recently studied "basics of machine learning" (chapter 5) in Deep Learning by Ian Goodfellow. They're explaining ML with most of statistical and probabilitical methods. But I wish to ...
1
vote
2answers
36 views

How can I explain this chart showing 5-days moving average?

I have plotted the frequency of items sold through time, trying to determine the trends by moving average. I considered a 5-days window. I would like to know if this approach makes sense and how I ...
1
vote
2answers
24 views

how can i compare two classification algorithms?

all, i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. ...
-1
votes
1answer
39 views

deep learning and uncertainty estimation

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML ...
0
votes
1answer
21 views

Use both differencing and normalization in time series modeling to make it stationary?

I am working on a time series dataset. Should we use both differencing and normalizing or either of the ones to make it stationary?
1
vote
0answers
35 views

Trusting p-values when errors are not normally distributed [duplicate]

Suppose I have a multiple linear regression model and the errors are not distributed normally.Does the central limit theorem hold true in this case? should i trust the p values of the coefficients of ...
4
votes
2answers
218 views

a data point - too little to spend resources during data preparation?

I am learning analytics online and have some quick questions. Usually when we do analysis, why is that we usually ignore the items/data points that are less frequent? Let's say for ex: we have is ...
0
votes
0answers
6 views

monte carlo simulation for impact to revenue by adjusting credit limits

I've been tasked with prioritizing projects for the year. One of them is to estimate the impact to net revenue if we were to adjust the purchase limits for our users. Our platform is transactional in ...
1
vote
1answer
22 views

Using BM25 to rank words

How effective is it to use BM25 to rank words, to be more specific i have a dictionary of words and i want to rank only words in a document that are also in my dictionary. I want to rank all words in ...
5
votes
2answers
96 views

Why is 10 considered the default value for k-fold cross-validation?

I understand very well what k-fold cross-validation is. In my studies, and at work, I've always heard something along the lines of: We most often use k=10 ...
0
votes
0answers
19 views

How to convert google trend relative values into more “absolute” values?

Searching key words or terms on google trends provides a time series of relative search numbers. I want to be able to analyse these trends more accurately compared to general trends in the same ...
3
votes
2answers
68 views

Distribution vs Histogram [closed]

I am not very good at understanding statistics jargon I tried to read articles on statistics every article says distribution and show a picture of bell curve. I understand that x axis of bell curve ...
1
vote
0answers
14 views

testing statistical significance when comparing regression models

I have 3 models, a random forest, a XGBoost and a baseline regression model. I've performed a 5-fold cross validation on all models and use MAE as the scoring metric. So this means I basically have 3 ...
0
votes
0answers
15 views

Sample size calculation for AB testing

I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My ...
0
votes
0answers
11 views

Treatment and Control selection in A/B Testing

I'm hoping to get a better understanding of A/B Testing design. In particular, I'm interested in understanding how treatment and control units are selected. I read that these 2 groups are selected ...
0
votes
1answer
22 views

How to adjust/smooth a certain number using constants or rules

Hi, I am handling a dataset with a customer purchase history. The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage. There are two ...
0
votes
0answers
8 views

Non-monotone missing data, and inverse probability weighting

I'm having difficulty identifying whether or not my missing data pattern is 'monotone'. I have two variables with missing data, and the missing data patterns in each variable do not completely ...

1
2 3 4 5
16