Questions tagged [statistics]
Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.
1,127
questions
0
votes
1
answer
19
views
Is there a measure to compare features on the basis of normality
I have a dataset of cars and it has many features including 'acceleration’, ‘horsepower’, and ‘mpg'.
I am supposed to check which if these features is the most similar to a normal distribution, so I ...
0
votes
0
answers
4
views
How to estimate the weights for a weighted average if not every weight is used in every example?
Say that you have a data set that contains $N$ examples. For each example, there are three main sections of important information. The first most important piece of information is the name of the ...
0
votes
0
answers
18
views
statistical analysis of two data sets have different number of sampling
I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
- 1
0
votes
0
answers
13
views
Generate evenly space point geography along a linestring
I have the total number of poles in a location (a sample area) and the road geometry (linestring) of each road in the area. I can calculate the average distance between poles, I want to distribute ...
- 1
0
votes
0
answers
23
views
How to calculate Pairwise Correlation Difference?
I was trying to implement the pairwise correlation difference (PCD) found in the following paper.
https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-00977-1
PCD measures the ...
0
votes
1
answer
41
views
Very basic but how to understand data statistically for machine learning?
I’m trying to solidify my statistics so I really know how to use them in my analysis/models. However my concept of statistical testing gets completely messed up by context. I’m unsure defining exactly ...
0
votes
0
answers
15
views
Alternative to partial dependence plots
In my understanding, to make the partial dependence plot, you select a feature of interest x and leave other features X unchanged, then for each example in the validation dataset you sample values in ...
0
votes
0
answers
15
views
Standardizing my target vs not standardizing
I've heard from multiple sources that it depends on whether I should standardize or not. Most of the time, people would say it doesn't make sense to do so, some would say it's better if I standardize ...
0
votes
0
answers
11
views
classification using simple relationships between time series data
I am looking to predict which courses are taught by which university professors at my school. More specifically, for each semester and professor I want to know the probability breakdown of which ...
- 101
1
vote
0
answers
21
views
Determining the start and stop of vehicle movement by GPS
There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
0
votes
0
answers
22
views
Is there anyway to evaluate the estimation results of least square
Consider the scenario where a practical problem is tackled utilizing the method of least squares. Upon each iteration, an estimation of the parameter $\theta$ is derived via $\hat{\theta} = (X^\top X)^...
0
votes
0
answers
25
views
How do I interpret a Scatter Plot of probabilities?
Help me with interpretation of the following scatter plot
1
vote
1
answer
38
views
Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?
I'm learning the Poisson distribution and am trying to "backward" calculate lambda if given a random value (k) and cumulative density function value (p). My k value is rather large, over 200,...
- 359
0
votes
1
answer
38
views
Workflow when making a machine learning model
I'm new to data science, and kinda confused with the workflow and steps to make a model. Before learning the math and concepts behind the algorithms like SVM, linear regressions, etc, I would just ...
1
vote
1
answer
64
views
Linear Regression and Logistic Regression
I'm a beginner, and I'm wondering whether a logistic regression in a nut-shell is just normalizing a linear regression? Correct me if I'm wrong, but I came to this conclusion because the predicted ...
0
votes
0
answers
4
views
Clustering or Finding Similarities Among Portfolio Allocations
I am trying to cluster a set of portfolios with percentage allocations among Stock, Bond, Other, and Cash. I am not sure what's the best way to go about this because the variables are interdependent ...
0
votes
1
answer
37
views
Suggestions to learn the Machine Learning models in greater depth?
I've been learning machine learning for the past few weeks from books and online courses. The books I've been reading, and currently still reading is "Hands-On Machine Learning with Scikit-Learn ...
0
votes
0
answers
8
views
How to add Lasso regularization to midas regression?
This is the code I am running now:
m17 <- midas_r(mls(bip_2018_2023_NA[0:60], 1:1, 1) ~mls(log_diff_load_seasonally_adjusted[0:240]^2, 4:8, 4)+ mls(log_diff_load_seasonally_adjusted[0:240], 4:8, 4),...
- 1
0
votes
0
answers
28
views
Prediction of the next number 0 to 9 that doesn't have a particular sequence
How to predict a number from 0 to 9 that are random and doesn't have any particular sequence that the numbers are following?
1
vote
0
answers
21
views
Which inferential test (T Test, F Test, ANOVA)
I have some sample data of customers and I am looking at the impact of consistent advertising (e.g. showing the same message) to see the impact of click through rates per session.
The analysis shows ...
0
votes
0
answers
10
views
What is the best sampling strategy for correlation analysis?
I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context.
Thanks
- 21
0
votes
0
answers
20
views
Bayesian Linear regression for non-gaussian distribution
Which distribution should be considered for non-gaussian distribution while building Bayesian Linear regression. I am trying to build this model using Pymc3 library but since I have just started to ...
0
votes
0
answers
16
views
Using standard deviation in the calculation of ratio to get the scale of values
In the youtube tutorial (Building makemore Part 3: Activations & Gradients, BatchNorm), standard deviation is always used to calculate ratio between two values (e.g. grad to data ratio, update to ...
- 1
0
votes
0
answers
8
views
How to model noisy time series data? Does non-linear modelling help?
Is it possible to model time series data that fluctuates. The main solution is to take first differences and make it easier to fit conventional models. What if non-linear models are built? Can they ...
- 1
0
votes
1
answer
223
views
Help with R - funModeling package no longer available, need help downloading older version
I have posted on GitHub but I'm not getting a response as quickly as I would like so I figured I'd seek help on multiple forums/websites. I am trying to do a few sample mean exercises from a textbook '...
- 101
0
votes
1
answer
57
views
Remove outlier from image array in Python
Could someone please suggest me what would be the best way to remove such huge number of outlier data from the image. The regular clipping between data range in numpy array would simply reduce the ...
- 101
0
votes
0
answers
5
views
Sales Comparison problem, how to apply Chi Square Test or alternatives?
Lets Say Before Pandemic, the Sales of Product A, B, ,C and D, are 780 , 350 , 250 , 100,
This is an Average of 36 Months , considering 12 months of Pandemic data is ignored, and after pandemic, 730 , ...
- 101
0
votes
1
answer
57
views
Prediction intervals for future timestamps - out-of-sample
I've created a model for out-of-sample forecasting that uses multistep recursive strategy to reduce my problem to regression, the predictions are sufficient but I was wondering if there is any ...
- 1
0
votes
0
answers
23
views
In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?
In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?
0
votes
1
answer
17
views
Unclear point in outlier construction based on an heavy-tail distribution
The following is an example from a book (An Introduction to Pattern Recognition and Machine Learning by P. Fieguth, page 417) on random variables and random vectors. Please consider the following.
...
0
votes
0
answers
23
views
Correlation in R between categorical and boolean variables
I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
- 1
0
votes
1
answer
40
views
how to evaluate a model on our data when the model is imported from a library and thus not trained by us?
The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
- 3
0
votes
1
answer
19
views
Approaches to dataset, whose elements have different size
I am working with a dataset where each elements is a square table of size m-by-n, where m (the number of rows) is the same for all the data points, while n (the number of columns) varies from tens to ...
- 101
0
votes
2
answers
30
views
Is probabilities mean of predicted class (RandomForest) a consistent estimator of class recall?
I'm working on a classification problem in order to predict among 50 different classes.
I'm using a Random Forest classifier and I'm using the predict_proba method ...
0
votes
1
answer
35
views
How can we distinguish meta-analysis and statistical analysis?
The two terms are frequently used in statistical methods. I find that statistical reasoning/ theory support meta analysis. What is real relation between two terms? Help me to understand the two ...
- 605
0
votes
1
answer
13
views
Bibliographic references on statistical analysis of classification results in machine learning
I am looking for a good must-have reference about statistical analysis of classification results in machine learning.
What books do you recommend about it?
0
votes
0
answers
14
views
Comparing multiple multivariate datasets
Take the two datasets below:
default rate
state
age
income
asofdate
10
Texas
55
100,000
202309
14
Texas
35
97,000
202309
18
Texas
55
95,000
202308
22
Texas
35
95,000
202308
8
New York
21
55,000
...
- 1
1
vote
1
answer
81
views
Which statistical procedure to use?
I'm new to Hypothesis testing. I can't really think of how to answer this -
Five students take a test one year and another test the next year. Their overall grades(in %) are given below for the test-1 ...
- 197
2
votes
2
answers
371
views
Parameter estimation in linear regression
Another test Q I couldn't answer -
We have marks of students belonging to 3 sections - A,B,C and two genders - M & F. Which regression model will not be able to estimate all the parameters?
1 ) ...
- 197
0
votes
0
answers
39
views
What is the distinction between clinical significance and statistical significance?
https://en.m.wikipedia.org/wiki/Clinical_significance
The Link says
"Jacobson-Truax is common method of calculating clinical significance. It involves calculating a Reliability Change Index (RCI)....
- 605
0
votes
0
answers
18
views
How to detect a kick signal
Background:
I am measuring the reflection signal from the pipe. First strong kick is called firing kick. The later small bumps ...
- 111
0
votes
2
answers
33
views
Lack of imagination when creating projects
Whenever I find a set of data to carry out a project, I end up running out of ideas and don't know how to proceed with the project. I don't know how to proceed or what information to look for. How do ...
0
votes
0
answers
19
views
Convert rank to percentage
If I have a series of counties ranked 1-95, 1 being the highest and 95 being the lowest and I want to convert that to percentage with 0 being the lowest and 100 being the highest. Would (1-(rank/95))*...
- 9
0
votes
1
answer
124
views
What is a good metric for comparing a single value to a distribution?
I'd like to compare single values from one distribution to another distribution, effectively transforming the second distribution in such way, that its values reflect both datasets simultaneously, i.e....
- 1
0
votes
1
answer
47
views
Creating well-separated bins
I want to distribute my hospital data into 4 tiers (bins) based on the score. Tier 1 (good hospitals) has the highest score and Tier 4 (not so good hospitals) has the lowest score. The tiers can be of ...
- 101
0
votes
0
answers
20
views
A hypercube with side length 1 in d dimensions is defined to be the set of points
The Question: A hypercube with side length 1 in d dimensions is defined to be the set of points (x1, x2, ..., xd) such that for all j = 1, 2, ..., d. The boundary of the hypercube is defined to be ...
0
votes
0
answers
14
views
Statistical aggregations for numerical features
Can someone explain why in a lot of machine learning competitions, many participants perform some statistical aggregations like mean and median aggregation for numerical features?
How can this improve ...
0
votes
0
answers
39
views
Statistical significance of classifier mean error rates
Given:
Error rates of 3 classifier types (CT) of performing 10 times a 5-fold cross validation
For each CT the mean error rate out of the 10*5=50 error rates is calculated
Goal:
Is the CT with the ...
- 1
0
votes
0
answers
32
views
How to split a range of numbers considering other variables as well?
Let's say that I have a vector of numbers and I'd like to split it into 3 most optimal ranges for example, then I suppose I can use k-means or Jenks natural breaks. If I'd like to do the same thing ...
0
votes
0
answers
15
views
How do you figure out how much of a user rating to attribute to each step?
I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this:
Run sentiment analysis on input
Run intent analysis on input
Choose gating weights ...
