Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Can an OCR model consistently recognize every digit of a long number correctly?

I'm working on OCR on scanned documents and we need to recognize the exact sequence of some printed numbers on it. Imagine you're reading a bank cheque serial number (16 digits) so the system needs to ...
TlifeProgram's user avatar
0 votes
0 answers
21 views

Why kind of data is appropriate for prediction and what kind is not? [closed]

What relationship between data is appropriate for predictive modeling and what type is not? How do we ensure we select the appropriate type of data for predictive modeling and not use data that is not ...
HelloDarkWorld's user avatar
0 votes
0 answers
17 views

Standard Error interpretation help

I performed a standard error on my machine learning model to predict protein structure. The graph Im showing here is a snippet of the actual data and I deleted some irrelevant info. The y axis is the ...
hypermiler3's user avatar
0 votes
0 answers
29 views

How to Determine the Minimum Value of a Continuous Variable for Predicting Categorical variable using Logistic Regression?

I am using logistic regression to predict df['MortSubiteCardiaque'], which contains 0 and 1, based on my continuous variable df['NTProBNP']. I would like to determine the threshold for df['NTProBNP'], ...
Mohamed kenani's user avatar
0 votes
0 answers
13 views

pyspark combined standard deviation?

I have a data frame which consists of multiple standard deviations with mean and count of each. my df is as follows ...
userDuR's user avatar
  • 101
0 votes
0 answers
15 views

Creating a project based "level of effort remaining" visualization for project tracking and management

I am looking to create a tracker to help manage the current and future progress of a large engineering project. The project is made up of many subprojects each with their own tracker. This is an excel ...
BRNZ's user avatar
  • 1
0 votes
0 answers
12 views

How to find the parameters of custom regression in python code?

I want to make a time series regression that has a custom formula lets say: Yt=Beta+1/(1+(some parameter sigma))Y(t-1)+(1/(1+(some parameter sigma))**2)Y(t-2)+Beta1X(t-1)+Beta2X(t-2) how to code this ...
J_Bake's user avatar
  • 1
13 votes
2 answers
556 views

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not? Data science seems to seem class imbalance as problematic and needing special techniques ...
Dave's user avatar
  • 3,698
0 votes
0 answers
25 views

Looking for a Statistical Modelling Technique for a Credibility Scoring Model

I’m in the process of developing a model that assigns a credibility score to fatigue reports within an organization. Employees can report feeling “tired” an unlimited number of times throughout the ...
Shibaprasad's user avatar
0 votes
1 answer
16 views

Univariate anomaly / outlier detection

I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection. I have a dataset of around 60K data points. Each data point is part ...
EyalG's user avatar
  • 1
0 votes
1 answer
50 views

Negative Log Likelihood of a Gaussian Model can be negative?

I am confused about the negative log likelihood of a guassian model: Can the negative likelihood of a gaussian model be negative ? lets suppose that the variance is going to 0 faster than the MSE ...
yoshcn's user avatar
  • 1
0 votes
0 answers
36 views

Unsupervised Learning with Time Series data?

I'm working with data which has daily aggregated user actvity over the course of several months (a user doesn't have to make an activity every day). I'm looking for a way to cluster similar users ...
user6132211's user avatar
0 votes
0 answers
9 views

Correlation in forecast error

I am working on a time series project and while calculating the prediction intervals using bootstrapping method I have to fulfill the assumptions of it. It has an assumption of uncorrelated forecast ...
Aakash Goyal's user avatar
0 votes
0 answers
12 views

ML development on data biased by historical treatments

I have a dataset where in each data point was subject to certain treatment (4 different treatments) in the past based on their riskiness. The riskiness was estimated by a logistic regression model in ...
NerdyMandy's user avatar
1 vote
0 answers
42 views

Occam's factor and the VC dimension

I was watching this lecture by Prof. Dr. Philipp Hennig (Probabilistic ML) and when he reached this formula which is the type two maximum log likelihood I had the following question: The Occam's ...
HAMDI ABDERRAHMENE's user avatar
0 votes
0 answers
40 views

Setting derivative to 0 when minimizing expected loss over possible data samples

Suppose, given $x\in \mathbb{R}^d$, you want $\theta$, which is the solution of $$ \text{argmin}_{\theta} \mathbb{E}_a[L(\theta;x,a)] $$ where $a \sim \mathcal{N}(x,\Sigma)$ and $L$ measures a (convex)...
user9781778's user avatar
1 vote
1 answer
23 views

Simple rebasing formula - beginner quesitons!

I have a bunch of risk scores out of 100 (there are a few 1000 of them). The issue is they all fall within 42 and 66 - but for the most part they are normally distributed (non-expert analysis). I was ...
George hill's user avatar
2 votes
1 answer
76 views

Multiple Hypothesis Testing in feature selection process

I am doing feature selection of features which are of binary nature i.e. each feature represents presence or absence of a substructure in a molecule. And I have a target variable of two classes. My ...
As13's user avatar
  • 75
0 votes
0 answers
23 views

How to estimate the weights for a weighted average if not every weight is used in every example?

Say that you have a data set that contains $N$ examples. For each example, there are three main sections of important information. The first most important piece of information is the name of the ...
Ignatius the Learner's user avatar
0 votes
0 answers
23 views

statistical analysis of two data sets have different number of sampling

I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
Fadi's user avatar
  • 1
0 votes
0 answers
15 views

Generate evenly space point geography along a linestring

I have the total number of poles in a location (a sample area) and the road geometry (linestring) of each road in the area. I can calculate the average distance between poles, I want to distribute ...
3XT's user avatar
  • 1
0 votes
0 answers
50 views

How to calculate Pairwise Correlation Difference?

I was trying to implement the pairwise correlation difference (PCD) found in the following paper. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-00977-1 PCD measures the ...
Sumon Jafrul's user avatar
0 votes
1 answer
46 views

Very basic but how to understand data statistically for machine learning?

I’m trying to solidify my statistics so I really know how to use them in my analysis/models. However my concept of statistical testing gets completely messed up by context. I’m unsure defining exactly ...
donutmonster's user avatar
0 votes
0 answers
21 views

Alternative to partial dependence plots

In my understanding, to make the partial dependence plot, you select a feature of interest x and leave other features X unchanged, then for each example in the validation dataset you sample values in ...
mindoverflow's user avatar
1 vote
0 answers
29 views

Standardizing my target versus not-standardizing

I've heard from multiple sources that it depends on whether I should standardize or not. Most of the time, people would say it doesn't make sense to do so, some would say it's better if I standardize ...
Justin Jonany's user avatar
0 votes
0 answers
12 views

classification using simple relationships between time series data

I am looking to predict which courses are taught by which university professors at my school. More specifically, for each semester and professor I want to know the probability breakdown of which ...
retep's user avatar
  • 101
1 vote
0 answers
29 views

Determining the start and stop of vehicle movement by GPS

There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
Abraham Lincoln's user avatar
0 votes
0 answers
24 views

Is there anyway to evaluate the estimation results of least square

Consider the scenario where a practical problem is tackled utilizing the method of least squares. Upon each iteration, an estimation of the parameter $\theta$ is derived via $\hat{\theta} = (X^\top X)^...
yangtzech's user avatar
0 votes
0 answers
27 views

How do I interpret a Scatter Plot of probabilities?

Help me with interpretation of the following scatter plot
Amina Issa's user avatar
1 vote
1 answer
44 views

Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?

I'm learning the Poisson distribution and am trying to "backward" calculate lambda if given a random value (k) and cumulative density function value (p). My k value is rather large, over 200,...
zipline86's user avatar
  • 359
0 votes
1 answer
72 views

Workflow when making a machine learning model

I'm new to data science, and kinda confused with the workflow and steps to make a model. Before learning the math and concepts behind the algorithms like SVM, linear regressions, etc, I would just ...
Justin Jonany's user avatar
1 vote
1 answer
72 views

Linear Regression and Logistic Regression

I'm a beginner, and I'm wondering whether a logistic regression in a nut-shell is just normalizing a linear regression? Correct me if I'm wrong, but I came to this conclusion because the predicted ...
Justin Jonany's user avatar
0 votes
0 answers
4 views

Clustering or Finding Similarities Among Portfolio Allocations

I am trying to cluster a set of portfolios with percentage allocations among Stock, Bond, Other, and Cash. I am not sure what's the best way to go about this because the variables are interdependent ...
Ahmad Bhatti's user avatar
0 votes
1 answer
42 views

Suggestions to learn the Machine Learning models in greater depth?

I've been learning machine learning for the past few weeks from books and online courses. The books I've been reading, and currently still reading is "Hands-On Machine Learning with Scikit-Learn ...
Justin Jonany's user avatar
0 votes
0 answers
109 views

Prediction of the next number 0 to 9 that doesn't have a particular sequence

How to predict a number from 0 to 9 that are random and doesn't have any particular sequence that the numbers are following?
user155498's user avatar
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
Imen F's user avatar
  • 21
0 votes
0 answers
24 views

Bayesian Linear regression for non-gaussian distribution

Which distribution should be considered for non-gaussian distribution while building Bayesian Linear regression. I am trying to build this model using Pymc3 library but since I have just started to ...
Archaeolexicologist's user avatar
0 votes
0 answers
21 views

Using standard deviation in the calculation of ratio to get the scale of values

In the youtube tutorial (Building makemore Part 3: Activations & Gradients, BatchNorm), standard deviation is always used to calculate ratio between two values (e.g. grad to data ratio, update to ...
Jayden's user avatar
  • 1
0 votes
0 answers
8 views

How to model noisy time series data? Does non-linear modelling help?

Is it possible to model time series data that fluctuates. The main solution is to take first differences and make it easier to fit conventional models. What if non-linear models are built? Can they ...
J_Bake's user avatar
  • 1
0 votes
1 answer
668 views

Help with R - funModeling package no longer available, need help downloading older version

I have posted on GitHub but I'm not getting a response as quickly as I would like so I figured I'd seek help on multiple forums/websites. I am trying to do a few sample mean exercises from a textbook '...
JohnColtraneisJC's user avatar
0 votes
1 answer
119 views

Remove outlier from image array in Python

Could someone please suggest me what would be the best way to remove such huge number of outlier data from the image. The regular clipping between data range in numpy array would simply reduce the ...
hillsonghimire's user avatar
0 votes
0 answers
6 views

Sales Comparison problem, how to apply Chi Square Test or alternatives?

Lets Say Before Pandemic, the Sales of Product A, B, ,C and D, are 780 , 350 , 250 , 100, This is an Average of 36 Months , considering 12 months of Pandemic data is ignored, and after pandemic, 730 , ...
user2458922's user avatar
0 votes
1 answer
79 views

Prediction intervals for future timestamps - out-of-sample

I've created a model for out-of-sample forecasting that uses multistep recursive strategy to reduce my problem to regression, the predictions are sufficient but I was wondering if there is any ...
kkkk0's user avatar
  • 1
0 votes
0 answers
23 views

In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?

In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?
user154739's user avatar
0 votes
1 answer
17 views

Unclear point in outlier construction based on an heavy-tail distribution

The following is an example from a book (An Introduction to Pattern Recognition and Machine Learning by P. Fieguth, page 417) on random variables and random vectors. Please consider the following. ...
user154739's user avatar
0 votes
0 answers
44 views

Correlation in R between categorical and boolean variables

I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
Max's user avatar
  • 1
0 votes
1 answer
40 views

how to evaluate a model on our data when the model is imported from a library and thus not trained by us?

The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
Shelby's user avatar
  • 3
0 votes
1 answer
20 views

Approaches to dataset, whose elements have different size

I am working with a dataset where each elements is a square table of size m-by-n, where m (the number of rows) is the same for all the data points, while n (the number of columns) varies from tens to ...
Roger V.'s user avatar
  • 103
0 votes
2 answers
36 views

Is probabilities mean of predicted class (RandomForest) a consistent estimator of class recall?

I'm working on a classification problem in order to predict among 50 different classes. I'm using a Random Forest classifier and I'm using the predict_proba method ...
Jerome X.'s user avatar
0 votes
1 answer
40 views

How can we distinguish meta-analysis and statistical analysis?

The two terms are frequently used in statistical methods. I find that statistical reasoning/ theory support meta analysis. What is real relation between two terms? Help me to understand the two ...
Subhash C. Davar's user avatar

1
2 3 4 5
23