# Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

1,121 questions
Filter by
Sorted by
Tagged with
1 vote
179 views

### How can you build a model based on non-independent imbalance data?

I am trying to predict customer churn based on the data that I have. I am defining churn as an activity that is not followed by another activity within a week. The customer might come back in two ...
7 views

### Can an OCR model consistently recognize every digit of a long number correctly?

I'm working on OCR on scanned documents and we need to recognize the exact sequence of some printed numbers on it. Imagine you're reading a bank cheque serial number (16 digits) so the system needs to ...
1 vote
116 views

### Handling Covariate Shift and Multi-collinearity in same Dataset

The problem is related to Regression problem. I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly ...
2k views

### Good t-SNE or bad t-SNE?

I have used tsne to visualise a large dataset and it has produced the following graph. I need help interpreting it, as I have never seen a tsne graph like this before! I am aware that not much ...
76 views

### What is the point of final test set in K-fold cross-validation?

I am carrying out logistic regression for my binary classification problem, and I have validated the model with kfold cross-validation (k=10). I don't understand why I need to have a final test set, ...
1 vote
81 views

### How to calculate lexical cohension and semantic informaticveness for a given dataset?

In 'Automatic construction of lexicons, taxonomies, ontologies, and other knowledge structures' they have mentioned; There are two slightly different classes of measure: lexical cohesion (sometimes ...
1 vote
6k views

### How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
661 views

### How is a coincidence matrix constructed for computing Krippendorff's alpha?

I am looking at two documents to help me learn about constructing coincidence matrices in order to gain a better understanding of Krippendorff's alpha. I am using these two: https://repository.upenn....
68 views

### How to forecast time bound contract based revenue?

I'm looking at ideas to see how I can forecast contract-based revenue. For example, I can have customers who have purchased a monthly mobile plan, an annual plan, or a 3-year plan. How can I use that ...
29 views

### How to Determine the Minimum Value of a Continuous Variable for Predicting Categorical variable using Logistic Regression?

I am using logistic regression to predict df['MortSubiteCardiaque'], which contains 0 and 1, based on my continuous variable df['NTProBNP']. I would like to determine the threshold for df['NTProBNP'], ...
21 views

### Why kind of data is appropriate for prediction and what kind is not? [closed]

What relationship between data is appropriate for predictive modeling and what type is not? How do we ensure we select the appropriate type of data for predictive modeling and not use data that is not ...
639 views

### A clear visualization of a two-way ANOVA

To provide a full yet simple picture of a 3-level, one-way ANOVA, I use the following visualization where variation within each group (the filled circles) and variation between the groups (black ...
79 views

### How a Random forest "learns" or How loss (objective function value) is propagated back so that a random forest can "Improve"?

Every Blog and Youtube video talks about the same steps: Choose that you have to build N number of tree and do the task 2-5 ...
13 views

### pyspark combined standard deviation?

I have a data frame which consists of multiple standard deviations with mean and count of each. my df is as follows ...
1 vote
212 views

### How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
17 views

### Standard Error interpretation help

I performed a standard error on my machine learning model to predict protein structure. The graph Im showing here is a snippet of the actual data and I deleted some irrelevant info. The y axis is the ...
123 views

1 vote
128 views

### How to calculate steady/incremental growth?

I have timeseries data for stocks at minute intervals. What is the best way to calculate incremental growth, for example if I have a stock's price from 9am to 2pm at minute intervals, how can calcule ...
82 views

### Statistical learning for data-limited systems

I'm currently conducting a review for quantitative methods being used for tropical inland fisheries. One of the major problems for modeling methods in tropical inland fisheries is the lack of data ...
15 views

### Creating a project based "level of effort remaining" visualization for project tracking and management

I am looking to create a tracker to help manage the current and future progress of a large engineering project. The project is made up of many subprojects each with their own tracker. This is an excel ...
38 views

### Why is the variance going down so much in this weight initialization problem(using pytorch)?

first look at this example >>> x = t.randn(512) >>> w = t.randn(512, 500000) >>> (x @ w).var() tensor(513.9548) it makes sense that ...
557 views

### Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not? Data science seems to seem class imbalance as problematic and needing special techniques ...
16 views

### Univariate anomaly / outlier detection

I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection. I have a dataset of around 60K data points. Each data point is part ...
393 views

### Feature Selection and Outlier Detection

How does feature selection impact outlier detection and also, removing outliers impact feature selection? It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance....
1 vote
108 views

### If the entire production dataset is known is advance, should I use it when fitting normalization parameters?

I found in multiple sources the recommendation not to fit the normalization parameters on the combined train/test dataset when evaluating the model, to prevent data leakage. I presume this ...
12 views

### How to find the parameters of custom regression in python code?

I want to make a time series regression that has a custom formula lets say: Yt=Beta+1/(1+(some parameter sigma))Y(t-1)+(1/(1+(some parameter sigma))**2)Y(t-2)+Beta1X(t-1)+Beta2X(t-2) how to code this ...
79 views

### Prediction intervals for future timestamps - out-of-sample

I've created a model for out-of-sample forecasting that uses multistep recursive strategy to reduce my problem to regression, the predictions are sufficient but I was wondering if there is any ...
1 vote
244 views

### A/B testing with non-Gaussian distributions

I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the ...
127 views

### Appropriate statistical test

I am working on a project where I have twitter user profiles and their tweets. The users are divided based on their number of followers in two groups (g1 and ...
1 vote
691 views

### How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
90 views

### Non IID variables and SVM Classifier

I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...