Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0 votes
1 answer
19 views

Is there a measure to compare features on the basis of normality

I have a dataset of cars and it has many features including 'acceleration’, ‘horsepower’, and ‘mpg'. I am supposed to check which if these features is the most similar to a normal distribution, so I ...
Tareq Ewaida's user avatar
0 votes
0 answers
4 views

How to estimate the weights for a weighted average if not every weight is used in every example?

Say that you have a data set that contains $N$ examples. For each example, there are three main sections of important information. The first most important piece of information is the name of the ...
Ignatius the Learner's user avatar
0 votes
0 answers
18 views

statistical analysis of two data sets have different number of sampling

I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
Fadi's user avatar
  • 1
0 votes
0 answers
13 views

Generate evenly space point geography along a linestring

I have the total number of poles in a location (a sample area) and the road geometry (linestring) of each road in the area. I can calculate the average distance between poles, I want to distribute ...
3XT's user avatar
  • 1
0 votes
0 answers
23 views

How to calculate Pairwise Correlation Difference?

I was trying to implement the pairwise correlation difference (PCD) found in the following paper. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-00977-1 PCD measures the ...
Sumon Jafrul's user avatar
0 votes
1 answer
41 views

Very basic but how to understand data statistically for machine learning?

I’m trying to solidify my statistics so I really know how to use them in my analysis/models. However my concept of statistical testing gets completely messed up by context. I’m unsure defining exactly ...
donutmonster's user avatar
0 votes
0 answers
15 views

Alternative to partial dependence plots

In my understanding, to make the partial dependence plot, you select a feature of interest x and leave other features X unchanged, then for each example in the validation dataset you sample values in ...
mindoverflow's user avatar
0 votes
0 answers
15 views

Standardizing my target vs not standardizing

I've heard from multiple sources that it depends on whether I should standardize or not. Most of the time, people would say it doesn't make sense to do so, some would say it's better if I standardize ...
Justin Jonany's user avatar
0 votes
0 answers
11 views

classification using simple relationships between time series data

I am looking to predict which courses are taught by which university professors at my school. More specifically, for each semester and professor I want to know the probability breakdown of which ...
retep's user avatar
  • 101
1 vote
0 answers
21 views

Determining the start and stop of vehicle movement by GPS

There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
Abraham Lincoln's user avatar
0 votes
0 answers
22 views

Is there anyway to evaluate the estimation results of least square

Consider the scenario where a practical problem is tackled utilizing the method of least squares. Upon each iteration, an estimation of the parameter $\theta$ is derived via $\hat{\theta} = (X^\top X)^...
yangtzech's user avatar
0 votes
0 answers
25 views

How do I interpret a Scatter Plot of probabilities?

Help me with interpretation of the following scatter plot
Amina Issa's user avatar
1 vote
1 answer
38 views

Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?

I'm learning the Poisson distribution and am trying to "backward" calculate lambda if given a random value (k) and cumulative density function value (p). My k value is rather large, over 200,...
zipline86's user avatar
  • 359
0 votes
1 answer
38 views

Workflow when making a machine learning model

I'm new to data science, and kinda confused with the workflow and steps to make a model. Before learning the math and concepts behind the algorithms like SVM, linear regressions, etc, I would just ...
Justin Jonany's user avatar
1 vote
1 answer
64 views

Linear Regression and Logistic Regression

I'm a beginner, and I'm wondering whether a logistic regression in a nut-shell is just normalizing a linear regression? Correct me if I'm wrong, but I came to this conclusion because the predicted ...
Justin Jonany's user avatar
0 votes
0 answers
4 views

Clustering or Finding Similarities Among Portfolio Allocations

I am trying to cluster a set of portfolios with percentage allocations among Stock, Bond, Other, and Cash. I am not sure what's the best way to go about this because the variables are interdependent ...
Ahmad Bhatti's user avatar
0 votes
1 answer
37 views

Suggestions to learn the Machine Learning models in greater depth?

I've been learning machine learning for the past few weeks from books and online courses. The books I've been reading, and currently still reading is "Hands-On Machine Learning with Scikit-Learn ...
Justin Jonany's user avatar
0 votes
0 answers
8 views

How to add Lasso regularization to midas regression?

This is the code I am running now: m17 <- midas_r(mls(bip_2018_2023_NA[0:60], 1:1, 1) ~mls(log_diff_load_seasonally_adjusted[0:240]^2, 4:8, 4)+ mls(log_diff_load_seasonally_adjusted[0:240], 4:8, 4),...
J_Bake's user avatar
  • 1
0 votes
0 answers
28 views

Prediction of the next number 0 to 9 that doesn't have a particular sequence

How to predict a number from 0 to 9 that are random and doesn't have any particular sequence that the numbers are following?
user155498's user avatar
1 vote
0 answers
21 views

Which inferential test (T Test, F Test, ANOVA)

I have some sample data of customers and I am looking at the impact of consistent advertising (e.g. showing the same message) to see the impact of click through rates per session. The analysis shows ...
TheJammyHamster's user avatar
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
Imen F's user avatar
  • 21
0 votes
0 answers
20 views

Bayesian Linear regression for non-gaussian distribution

Which distribution should be considered for non-gaussian distribution while building Bayesian Linear regression. I am trying to build this model using Pymc3 library but since I have just started to ...
Archaeolexicologist's user avatar
0 votes
0 answers
16 views

Using standard deviation in the calculation of ratio to get the scale of values

In the youtube tutorial (Building makemore Part 3: Activations & Gradients, BatchNorm), standard deviation is always used to calculate ratio between two values (e.g. grad to data ratio, update to ...
Jayden's user avatar
  • 1
0 votes
0 answers
8 views

How to model noisy time series data? Does non-linear modelling help?

Is it possible to model time series data that fluctuates. The main solution is to take first differences and make it easier to fit conventional models. What if non-linear models are built? Can they ...
J_Bake's user avatar
  • 1
0 votes
1 answer
223 views

Help with R - funModeling package no longer available, need help downloading older version

I have posted on GitHub but I'm not getting a response as quickly as I would like so I figured I'd seek help on multiple forums/websites. I am trying to do a few sample mean exercises from a textbook '...
JohnColtraneisJC's user avatar
0 votes
1 answer
57 views

Remove outlier from image array in Python

Could someone please suggest me what would be the best way to remove such huge number of outlier data from the image. The regular clipping between data range in numpy array would simply reduce the ...
hillsonghimire's user avatar
0 votes
0 answers
5 views

Sales Comparison problem, how to apply Chi Square Test or alternatives?

Lets Say Before Pandemic, the Sales of Product A, B, ,C and D, are 780 , 350 , 250 , 100, This is an Average of 36 Months , considering 12 months of Pandemic data is ignored, and after pandemic, 730 , ...
user2458922's user avatar
0 votes
1 answer
57 views

Prediction intervals for future timestamps - out-of-sample

I've created a model for out-of-sample forecasting that uses multistep recursive strategy to reduce my problem to regression, the predictions are sufficient but I was wondering if there is any ...
kkkk0's user avatar
  • 1
0 votes
0 answers
23 views

In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?

In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?
user154739's user avatar
0 votes
1 answer
17 views

Unclear point in outlier construction based on an heavy-tail distribution

The following is an example from a book (An Introduction to Pattern Recognition and Machine Learning by P. Fieguth, page 417) on random variables and random vectors. Please consider the following. ...
user154739's user avatar
0 votes
0 answers
23 views

Correlation in R between categorical and boolean variables

I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
Max's user avatar
  • 1
0 votes
1 answer
40 views

how to evaluate a model on our data when the model is imported from a library and thus not trained by us?

The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
Shelby's user avatar
  • 3
0 votes
1 answer
19 views

Approaches to dataset, whose elements have different size

I am working with a dataset where each elements is a square table of size m-by-n, where m (the number of rows) is the same for all the data points, while n (the number of columns) varies from tens to ...
Roger V.'s user avatar
  • 101
0 votes
2 answers
30 views

Is probabilities mean of predicted class (RandomForest) a consistent estimator of class recall?

I'm working on a classification problem in order to predict among 50 different classes. I'm using a Random Forest classifier and I'm using the predict_proba method ...
Jerome X.'s user avatar
0 votes
1 answer
35 views

How can we distinguish meta-analysis and statistical analysis?

The two terms are frequently used in statistical methods. I find that statistical reasoning/ theory support meta analysis. What is real relation between two terms? Help me to understand the two ...
Subhash C. Davar's user avatar
0 votes
1 answer
13 views

Bibliographic references on statistical analysis of classification results in machine learning

I am looking for a good must-have reference about statistical analysis of classification results in machine learning. What books do you recommend about it?
user154333's user avatar
0 votes
0 answers
14 views

Comparing multiple multivariate datasets

Take the two datasets below: default rate state age income asofdate 10 Texas 55 100,000 202309 14 Texas 35 97,000 202309 18 Texas 55 95,000 202308 22 Texas 35 95,000 202308 8 New York 21 55,000 ...
rpiston's user avatar
1 vote
1 answer
81 views

Which statistical procedure to use?

I'm new to Hypothesis testing. I can't really think of how to answer this - Five students take a test one year and another test the next year. Their overall grades(in %) are given below for the test-1 ...
a_jelly_fish's user avatar
2 votes
2 answers
371 views

Parameter estimation in linear regression

Another test Q I couldn't answer - We have marks of students belonging to 3 sections - A,B,C and two genders - M & F. Which regression model will not be able to estimate all the parameters? 1 ) ...
a_jelly_fish's user avatar
0 votes
0 answers
39 views

What is the distinction between clinical significance and statistical significance?

https://en.m.wikipedia.org/wiki/Clinical_significance The Link says "Jacobson-Truax is common method of calculating clinical significance. It involves calculating a Reliability Change Index (RCI)....
Subhash C. Davar's user avatar
0 votes
0 answers
18 views

How to detect a kick signal

Background: I am measuring the reflection signal from the pipe. First strong kick is called firing kick. The later small bumps ...
ii2's user avatar
  • 111
0 votes
2 answers
33 views

Lack of imagination when creating projects

Whenever I find a set of data to carry out a project, I end up running out of ideas and don't know how to proceed with the project. I don't know how to proceed or what information to look for. How do ...
Davi Komura's user avatar
0 votes
0 answers
19 views

Convert rank to percentage

If I have a series of counties ranked 1-95, 1 being the highest and 95 being the lowest and I want to convert that to percentage with 0 being the lowest and 100 being the highest. Would (1-(rank/95))*...
Mike's user avatar
  • 9
0 votes
1 answer
124 views

What is a good metric for comparing a single value to a distribution?

I'd like to compare single values from one distribution to another distribution, effectively transforming the second distribution in such way, that its values reflect both datasets simultaneously, i.e....
Hayes's user avatar
  • 1
0 votes
1 answer
47 views

Creating well-separated bins

I want to distribute my hospital data into 4 tiers (bins) based on the score. Tier 1 (good hospitals) has the highest score and Tier 4 (not so good hospitals) has the lowest score. The tiers can be of ...
museshad's user avatar
  • 101
0 votes
0 answers
20 views

A hypercube with side length 1 in d dimensions is defined to be the set of points

The Question: A hypercube with side length 1 in d dimensions is defined to be the set of points (x1, x2, ..., xd) such that for all j = 1, 2, ..., d. The boundary of the hypercube is defined to be ...
SilianRail's user avatar
0 votes
0 answers
14 views

Statistical aggregations for numerical features

Can someone explain why in a lot of machine learning competitions, many participants perform some statistical aggregations like mean and median aggregation for numerical features? How can this improve ...
Mohamed7894's user avatar
0 votes
0 answers
39 views

Statistical significance of classifier mean error rates

Given: Error rates of 3 classifier types (CT) of performing 10 times a 5-fold cross validation For each CT the mean error rate out of the 10*5=50 error rates is calculated Goal: Is the CT with the ...
Max's user avatar
  • 1
0 votes
0 answers
32 views

How to split a range of numbers considering other variables as well?

Let's say that I have a vector of numbers and I'd like to split it into 3 most optimal ranges for example, then I suppose I can use k-means or Jenks natural breaks. If I'd like to do the same thing ...
user152274's user avatar
0 votes
0 answers
15 views

How do you figure out how much of a user rating to attribute to each step?

I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this: Run sentiment analysis on input Run intent analysis on input Choose gating weights ...
TheEnvironmentalist's user avatar

1
2 3 4 5
23