Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
1 vote
1 answer
179 views

How can you build a model based on non-independent imbalance data?

I am trying to predict customer churn based on the data that I have. I am defining churn as an activity that is not followed by another activity within a week. The customer might come back in two ...
0 votes
0 answers
7 views

Can an OCR model consistently recognize every digit of a long number correctly?

I'm working on OCR on scanned documents and we need to recognize the exact sequence of some printed numbers on it. Imagine you're reading a bank cheque serial number (16 digits) so the system needs to ...
1 vote
1 answer
116 views

Handling Covariate Shift and Multi-collinearity in same Dataset

The problem is related to Regression problem. I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly ...
2 votes
2 answers
2k views

Good t-SNE or bad t-SNE?

I have used tsne to visualise a large dataset and it has produced the following graph. I need help interpreting it, as I have never seen a tsne graph like this before! I am aware that not much ...
2 votes
1 answer
76 views

What is the point of final test set in K-fold cross-validation?

I am carrying out logistic regression for my binary classification problem, and I have validated the model with kfold cross-validation (k=10). I don't understand why I need to have a final test set, ...
1 vote
1 answer
81 views

How to calculate lexical cohension and semantic informaticveness for a given dataset?

In 'Automatic construction of lexicons, taxonomies, ontologies, and other knowledge structures' they have mentioned; There are two slightly different classes of measure: lexical cohesion (sometimes ...
1 vote
2 answers
6k views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
2 votes
1 answer
661 views

How is a coincidence matrix constructed for computing Krippendorff's alpha?

I am looking at two documents to help me learn about constructing coincidence matrices in order to gain a better understanding of Krippendorff's alpha. I am using these two: https://repository.upenn....
0 votes
1 answer
68 views

How to forecast time bound contract based revenue?

I'm looking at ideas to see how I can forecast contract-based revenue. For example, I can have customers who have purchased a monthly mobile plan, an annual plan, or a 3-year plan. How can I use that ...
0 votes
0 answers
29 views

How to Determine the Minimum Value of a Continuous Variable for Predicting Categorical variable using Logistic Regression?

I am using logistic regression to predict df['MortSubiteCardiaque'], which contains 0 and 1, based on my continuous variable df['NTProBNP']. I would like to determine the threshold for df['NTProBNP'], ...
0 votes
0 answers
21 views

Why kind of data is appropriate for prediction and what kind is not? [closed]

What relationship between data is appropriate for predictive modeling and what type is not? How do we ensure we select the appropriate type of data for predictive modeling and not use data that is not ...
10 votes
1 answer
639 views

A clear visualization of a two-way ANOVA

To provide a full yet simple picture of a 3-level, one-way ANOVA, I use the following visualization where variation within each group (the filled circles) and variation between the groups (black ...
0 votes
2 answers
79 views

How a Random forest "learns" or How loss (objective function value) is propagated back so that a random forest can "Improve"?

Every Blog and Youtube video talks about the same steps: Choose that you have to build N number of tree and do the task 2-5 ...
0 votes
0 answers
13 views

pyspark combined standard deviation?

I have a data frame which consists of multiple standard deviations with mean and count of each. my df is as follows ...
1 vote
1 answer
212 views

How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
0 votes
0 answers
17 views

Standard Error interpretation help

I performed a standard error on my machine learning model to predict protein structure. The graph Im showing here is a snippet of the actual data and I deleted some irrelevant info. The y axis is the ...
2 votes
1 answer
123 views

Adding high p-value and low R square features in linear regression model to improve result

I am working on a linear regression problem. The features for my analysis have been selected using p-values and domain knowledge. After selecting these features, the performance of $R^2$ and the $...
1 vote
1 answer
397 views

How to find average lag time with variance & confidence of two time series

I have two variables as time series, one a consequent of the other, I would like to find the average time delay it takes the dependent variable to act on the independent variable. Additionally, I ...
0 votes
3 answers
234 views

ML, Statistics and Mathematics

I have just started getting my hands wet in ML and every time I try delving deeper into the concepts/code, I face the challenges of the mathematics and its cryptic notations. Coming from a Computer ...
0 votes
1 answer
99 views

Confusion on Outliers

I am not able to distinguish outliers: when to go with the std. dev. or when we need to go with the median. My understanding on std. dev. is: if the data point is away from the mean by more than 2 std....
2 votes
3 answers
1k views

A/B Testing (Binomial Distribution vs Random Distribution)

When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
6 votes
1 answer
183 views

Predicting change of shapes/coordinates

I'm trying to find a way to predict/calculate how a shape (e.g. outline of a glacier) will change in the future—based on its history (previous shape) and additional factors (e.g. Δtemperature). In my ...
1 vote
1 answer
627 views

Sampling methods for Text datasets (NLP)

I am working on two text datasets, one is having 68k text samples and other is having 100k text samples. I have encoded the text datasets into bert embedding. ...
0 votes
2 answers
581 views

Alternating Least Squares

can anyone explain difference between Alternating Least Squares(ALS) and the recommendation systems? it will be helpful if you give me an example.
5 votes
1 answer
710 views

Linear Regression bad results after log transformation

I have a dataset that has the following columns: The variable I'm trying to predict is "rent". My dataset looks a lot similar to what happens in this notebook. I tried to normalize the rent ...
0 votes
1 answer
69 views

Is there a safe and simple way to estimate a standard deviation for a next subset?

In case I receive only standard deviation from a sensor of a value $v$ (that is btw normally distributed) each 4th minute but need to provide a standard deviation $\sigma$ for each 15 minutes is there ...
1 vote
1 answer
145 views

Relationships between groups of features against independent variables

I have several groups of features that I'd like to test against independent variables. The idea is to find which groups tend to be associated with a specific value of an independent variable. Let's ...
2 votes
1 answer
1k views

How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I ...
0 votes
3 answers
385 views

When R2 score and MSE are not correlated

I'm training some forecasting models and then, to check performance I see several metrics. It's surprising for me when they are no related, for example: Let's suppose I'd have two models, A and B. --&...
2 votes
1 answer
143 views

Store's unseen items sales forecasting

I am working on sales forecasting problem.I am able to provide data about which items got sold and not sold to the algorithm.How to provide algorithm information about items that are not present in ...
0 votes
1 answer
685 views

Population stability Index vs Population Accuracy Index

Can anyone explain to me the difference between Population Stability Index(PSI) and Population Accuracy Index(PAI)?
4 votes
2 answers
612 views

Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
1 vote
1 answer
143 views

Replacing count by percentage in R

I have the following code : ...
5 votes
1 answer
252 views

What is the advantage of using Euler's number ($e^x$) instead of another base in the softmax equation?

I understand the softmax equation is $\boldsymbol{P}(y=j \mid x)=\frac{e^{x_{j}}}{\sum_{k=1}^{K} e^{x_{k}}}$ My question is: why use $e^x$ instead of say, $3^x$. I understand $e^x$ is it's own ...
1 vote
3 answers
128 views

How to calculate steady/incremental growth?

I have timeseries data for stocks at minute intervals. What is the best way to calculate incremental growth, for example if I have a stock's price from 9am to 2pm at minute intervals, how can calcule ...
3 votes
2 answers
82 views

Statistical learning for data-limited systems

I'm currently conducting a review for quantitative methods being used for tropical inland fisheries. One of the major problems for modeling methods in tropical inland fisheries is the lack of data ...
0 votes
0 answers
15 views

Creating a project based "level of effort remaining" visualization for project tracking and management

I am looking to create a tracker to help manage the current and future progress of a large engineering project. The project is made up of many subprojects each with their own tracker. This is an excel ...
0 votes
2 answers
38 views

Why is the variance going down so much in this weight initialization problem(using pytorch)?

first look at this example >>> x = t.randn(512) >>> w = t.randn(512, 500000) >>> (x @ w).var() tensor(513.9548) it makes sense that ...
13 votes
2 answers
557 views

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not? Data science seems to seem class imbalance as problematic and needing special techniques ...
0 votes
1 answer
16 views

Univariate anomaly / outlier detection

I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection. I have a dataset of around 60K data points. Each data point is part ...
2 votes
1 answer
393 views

Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
1 vote
2 answers
108 views

If the entire production dataset is known is advance, should I use it when fitting normalization parameters?

I found in multiple sources the recommendation not to fit the normalization parameters on the combined train/test dataset when evaluating the model, to prevent data leakage. I presume this ...
0 votes
0 answers
12 views

How to find the parameters of custom regression in python code?

I want to make a time series regression that has a custom formula lets say: Yt=Beta+1/(1+(some parameter sigma))Y(t-1)+(1/(1+(some parameter sigma))**2)Y(t-2)+Beta1X(t-1)+Beta2X(t-2) how to code this ...
0 votes
1 answer
79 views

Prediction intervals for future timestamps - out-of-sample

I've created a model for out-of-sample forecasting that uses multistep recursive strategy to reduce my problem to regression, the predictions are sufficient but I was wondering if there is any ...
1 vote
3 answers
244 views

A/B testing with non-Gaussian distributions

I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the ...
0 votes
1 answer
127 views

Appropriate statistical test

I am working on a project where I have twitter user profiles and their tweets. The users are divided based on their number of followers in two groups (g1 and ...
1 vote
3 answers
691 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
0 votes
1 answer
90 views

Non IID variables and SVM Classifier

I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
4 votes
2 answers
777 views

How do I know that model performance improvement is significant?

Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one ...
1 vote
1 answer
234 views

statistical significance test between binary label features

I have 667 features and I want to find features that have a significant boundary between a binary class label before I apply a classification model (e.g Naive Bayes/ SVM) to improve classification ...

1
2 3 4 5
23