Skip to main content

Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Forecasting Resource Depletion in a Distributed System

I manage a distributed system where each node contains six interchangeable resource slots, sourced from a diverse pool of resource types. Each type has a finite number of units, which get consumed ...
reuseman's user avatar
  • 101
0 votes
0 answers
42 views

When is sampling bias acceptable?

Overview: Dataset is small and a bit messy and the task is to classify 5 classes wherein the targets are ordinal. Feature Engineering and Selection, Model Tuning, etc. did not produce acceptable ...
easymoneysniper's user avatar
0 votes
0 answers
19 views

Adaptive Lasso Coefficient Weights

I'm trying to understand how the Adaptive part of Adaptive Lasso works. I understand that theoretically, the weights for zero coefficients are inflated to infinity. But can someone explain this ...
user162172's user avatar
0 votes
0 answers
16 views

Variable Selection and model prediction

In a supervised problem, I used randomForest for variable selection to identify the most important features. Question: am I required to use a random forest model for subsequent predictions, or can I ...
Zakaria Faouzi's user avatar
0 votes
0 answers
6 views

Synthetic Control for intermittent treatment

I'm exploring the application of the synthetic control method (SCM) to analyze the impact of intermittent treatments or interventions on a target outcome. I'm interested in understanding how SCM can ...
Notorious's user avatar
0 votes
0 answers
13 views

How to get probability of an outcome from skewed t distribution in R

I am trying to calculate the probability of stock return to be greater than X in next 28 days, using the skewed t-distribution as it fits the best to the ...
Krishnang K Dalal's user avatar
-1 votes
0 answers
14 views

Which statistical analysis method to use, and what would be the outcomes

I have an assignment to do, no calculations needed but would need to decide what to use and how I would run it. Data from 2 questions: "Select up three personality traits that you would find most ...
Peter's user avatar
  • 1
0 votes
0 answers
9 views

Can we calculate Bayes Error rate, if we have a simulated data?

I am going through ISL(Python) and in section 2.2.3 ( Page No. 36), the author writes, "For our simulated data, the Bayes error is 0.133. It is greater than zero, because the classes overlap in ...
Prashant Kumar's user avatar
0 votes
0 answers
17 views

Calculation of Covariance Matrices for a QDA classifier in Python Numpy

For a school project, I have to design a QDA classifier for 28x28 pixel images of letters in sign language. I have been given 27455 images for training, which have to be flattened to a 784 pixel ...
Neev Penkar's user avatar
0 votes
0 answers
15 views

VC dimension of a class H that assigns 1 to at most k points or that assigns 0 to at most k points

Let $X$ be a finite domain and $k$ a number such that $k\le|X|$. Consider the hypothesis class $$ H:=\{h:|\{x\in X:h(x)=1\}|\le k\text{ or }|\{x\in X:h(x)=0\}|\le k\}. $$ Find the VC dimension of $H$.
harshchy2210's user avatar
2 votes
1 answer
36 views

Time Series Analysis and Price Elasticity

Introduction: As of now, I am a fourth year data science student. As of now, I also have my own company where I work parttime (8/12 hours per week) to gain some more experience in the domain. As you ...
Martijn's user avatar
  • 21
0 votes
0 answers
21 views

Question on theory from original GBM article

I am reading the original gradient boosting machine article and, maybe because my statistics are a bit rusty, have a few questions on one section. In section ...
CarterKF's user avatar
0 votes
0 answers
20 views

Continuous Function from Binned Data with Consistent Integrals

I have binned energy production data (in Wh) in 5-minute intervals. This means that, at the timestamp of t + 5 minutes, the value represents the amount of energy generated in the interval from t to t +...
Benjamin K's user avatar
0 votes
0 answers
10 views

Need to put my formula into a graph in R

...
T15266489's user avatar
0 votes
0 answers
19 views

Generating a Sample T-Distribution in Python

I'm new to DataScience. I'm going through the stats module and the ways to create a sample T-distribution. Using the below line to generate the sample. However, my question is how is the array ...
naveent2785's user avatar
0 votes
0 answers
7 views

How to calcualte LR_statistic for the below data

I am trying to calculate LR_statistic for two models where my log-likelihood ratio values as follows. model1.llf = -2326.9 and model2.llf = -4649.1. When read ...
udaykumar gajavalli's user avatar
0 votes
0 answers
18 views

Am I suppose to do any statistical test on results?

I am doing an email campaign. Before sending emails to users I divided my user base into treatment and control groups (50-50). Divided the groups in such a way that no difference in user behaviour. I ...
udaykumar gajavalli's user avatar
2 votes
0 answers
18 views

Best practices for handling "NA" when all NA values exist due to being below the limit of detection?

I am working in R, and have a data set which has a few metabolite concentration values as continuous variables. Anywhere that the concentration was too low to be detected it simply says <LOD. This ...
KLN-RDN's user avatar
  • 21
0 votes
0 answers
23 views

Find out Statistical test for binary data

we are ran an email campaign for users which is called my treatment and there is control group as well. We want to measure the performance using a Metric called A. If he belongs to metric A then we ...
udaykumar gajavalli's user avatar
0 votes
0 answers
13 views

Can an OCR model consistently recognize every digit of a long number correctly?

I'm working on OCR on scanned documents and we need to recognize the exact sequence of some printed numbers on it. Imagine you're reading a bank cheque serial number (16 digits) so the system needs to ...
TlifeProgram's user avatar
1 vote
0 answers
33 views

Standard Error interpretation help

I performed a standard error on my machine learning model to predict protein structure. The graph Im showing here is a snippet of the actual data and I deleted some irrelevant info. The y axis is the ...
hypermiler3's user avatar
0 votes
0 answers
30 views

How to Determine the Minimum Value of a Continuous Variable for Predicting Categorical variable using Logistic Regression?

I am using logistic regression to predict df['MortSubiteCardiaque'], which contains 0 and 1, based on my continuous variable df['NTProBNP']. I would like to determine the threshold for df['NTProBNP'], ...
Mohamed kenani's user avatar
0 votes
0 answers
14 views

pyspark combined standard deviation?

I have a data frame which consists of multiple standard deviations with mean and count of each. my df is as follows ...
userDuR's user avatar
  • 101
0 votes
0 answers
15 views

Creating a project based "level of effort remaining" visualization for project tracking and management

I am looking to create a tracker to help manage the current and future progress of a large engineering project. The project is made up of many subprojects each with their own tracker. This is an excel ...
BRNZ's user avatar
  • 1
0 votes
0 answers
12 views

How to find the parameters of custom regression in python code?

I want to make a time series regression that has a custom formula lets say: Yt=Beta+1/(1+(some parameter sigma))Y(t-1)+(1/(1+(some parameter sigma))**2)Y(t-2)+Beta1X(t-1)+Beta2X(t-2) how to code this ...
J_Bake's user avatar
  • 1
14 votes
2 answers
609 views

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not? Data science seems to seem class imbalance as problematic and needing special techniques ...
Dave's user avatar
  • 3,863
0 votes
0 answers
25 views

Looking for a Statistical Modelling Technique for a Credibility Scoring Model

I’m in the process of developing a model that assigns a credibility score to fatigue reports within an organization. Employees can report feeling “tired” an unlimited number of times throughout the ...
Shibaprasad's user avatar
0 votes
1 answer
18 views

Univariate anomaly / outlier detection

I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection. I have a dataset of around 60K data points. Each data point is part ...
EyalG's user avatar
  • 1
0 votes
1 answer
89 views

Negative Log Likelihood of a Gaussian Model can be negative?

I am confused about the negative log likelihood of a guassian model: Can the negative likelihood of a gaussian model be negative ? lets suppose that the variance is going to 0 faster than the MSE ...
yoshcn's user avatar
  • 1
0 votes
0 answers
43 views

Unsupervised Learning with Time Series data?

I'm working with data which has daily aggregated user actvity over the course of several months (a user doesn't have to make an activity every day). I'm looking for a way to cluster similar users ...
user6132211's user avatar
0 votes
0 answers
9 views

Correlation in forecast error

I am working on a time series project and while calculating the prediction intervals using bootstrapping method I have to fulfill the assumptions of it. It has an assumption of uncorrelated forecast ...
Aakash Goyal's user avatar
0 votes
0 answers
12 views

ML development on data biased by historical treatments

I have a dataset where in each data point was subject to certain treatment (4 different treatments) in the past based on their riskiness. The riskiness was estimated by a logistic regression model in ...
NerdyMandy's user avatar
1 vote
0 answers
42 views

Occam's factor and the VC dimension

I was watching this lecture by Prof. Dr. Philipp Hennig (Probabilistic ML) and when he reached this formula which is the type two maximum log likelihood I had the following question: The Occam's ...
HAMDI ABDERRAHMENE's user avatar
0 votes
0 answers
40 views

Setting derivative to 0 when minimizing expected loss over possible data samples

Suppose, given $x\in \mathbb{R}^d$, you want $\theta$, which is the solution of $$ \text{argmin}_{\theta} \mathbb{E}_a[L(\theta;x,a)] $$ where $a \sim \mathcal{N}(x,\Sigma)$ and $L$ measures a (convex)...
user9781778's user avatar
1 vote
1 answer
23 views

Simple rebasing formula - beginner quesitons!

I have a bunch of risk scores out of 100 (there are a few 1000 of them). The issue is they all fall within 42 and 66 - but for the most part they are normally distributed (non-expert analysis). I was ...
George hill's user avatar
2 votes
1 answer
85 views

Multiple Hypothesis Testing in feature selection process

I am doing feature selection of features which are of binary nature i.e. each feature represents presence or absence of a substructure in a molecule. And I have a target variable of two classes. My ...
As13's user avatar
  • 77
0 votes
0 answers
25 views

How to estimate the weights for a weighted average if not every weight is used in every example?

Say that you have a data set that contains $N$ examples. For each example, there are three main sections of important information. The first most important piece of information is the name of the ...
Ignatius the Learner's user avatar
0 votes
0 answers
24 views

statistical analysis of two data sets have different number of sampling

I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
Fadi's user avatar
  • 1
0 votes
0 answers
15 views

Generate evenly space point geography along a linestring

I have the total number of poles in a location (a sample area) and the road geometry (linestring) of each road in the area. I can calculate the average distance between poles, I want to distribute ...
3XT's user avatar
  • 1
0 votes
0 answers
70 views

How to calculate Pairwise Correlation Difference?

I was trying to implement the pairwise correlation difference (PCD) found in the following paper. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-00977-1 PCD measures the ...
Sumon Jafrul's user avatar
0 votes
1 answer
46 views

Very basic but how to understand data statistically for machine learning?

I’m trying to solidify my statistics so I really know how to use them in my analysis/models. However my concept of statistical testing gets completely messed up by context. I’m unsure defining exactly ...
donutmonster's user avatar
0 votes
0 answers
25 views

Alternative to partial dependence plots

In my understanding, to make the partial dependence plot, you select a feature of interest x and leave other features X unchanged, then for each example in the validation dataset you sample values in ...
mindoverflow's user avatar
1 vote
0 answers
29 views

Standardizing my target versus not-standardizing

I've heard from multiple sources that it depends on whether I should standardize or not. Most of the time, people would say it doesn't make sense to do so, some would say it's better if I standardize ...
Justin Jonany's user avatar
0 votes
0 answers
12 views

classification using simple relationships between time series data

I am looking to predict which courses are taught by which university professors at my school. More specifically, for each semester and professor I want to know the probability breakdown of which ...
retep's user avatar
  • 101
1 vote
0 answers
30 views

Determining the start and stop of vehicle movement by GPS

There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
Abraham Lincoln's user avatar
0 votes
0 answers
24 views

Is there anyway to evaluate the estimation results of least square

Consider the scenario where a practical problem is tackled utilizing the method of least squares. Upon each iteration, an estimation of the parameter $\theta$ is derived via $\hat{\theta} = (X^\top X)^...
yangtzech's user avatar
0 votes
0 answers
27 views

How do I interpret a Scatter Plot of probabilities?

Help me with interpretation of the following scatter plot
Amina Issa's user avatar
1 vote
1 answer
57 views

Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?

I'm learning the Poisson distribution and am trying to "backward" calculate lambda if given a random value (k) and cumulative density function value (p). My k value is rather large, over 200,...
zipline86's user avatar
  • 359
0 votes
1 answer
93 views

Workflow when making a machine learning model

I'm new to data science, and kinda confused with the workflow and steps to make a model. Before learning the math and concepts behind the algorithms like SVM, linear regressions, etc, I would just ...
Justin Jonany's user avatar
1 vote
1 answer
72 views

Linear Regression and Logistic Regression

I'm a beginner, and I'm wondering whether a logistic regression in a nut-shell is just normalizing a linear regression? Correct me if I'm wrong, but I came to this conclusion because the predicted ...
Justin Jonany's user avatar

1
2 3 4 5
23