Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Normalise the scores by small and large numbers division

I am working on a scoring problem use-case where I will score each task based on few aspects and take an average of scores of tasks in each experiment. For each experiment I will have 1 to N number of ...
0
votes
0answers
9 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
1
vote
0answers
15 views

Compare ranking across trials with pandas

I am carrying out several runs of an experiment with different configurations on many different inputs. The experiment has a random element and I'm trying to investigate whether this randomness ...
0
votes
1answer
24 views

I want to run PCA on a data set that will be aggregated by country. Should I aggregate the data before or after I standardize the data, and why?

Basically the title asks my question. I have the results of a survey that was filled out by people from different countries. I have been asked to analyze the data using PCA and see what findings I can ...
0
votes
0answers
15 views

KL divergence for exponential family distribution

In reinforcement learning, normal distribution is commonly used for continuous actions. I'm checking Pytorch's implementation for KL divergence between two normal distributions. I know it's impossible ...
1
vote
2answers
48 views

How to build a model on a dataset having 40% missing values in most of the variables?

I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category)...
0
votes
0answers
13 views

How to compare between two datasets of lexical density?

I have two dataset from two different texts representing lexical density as a proportion based on a corpus. Both datasets are represented in the images below. Now, let's suppose I want to know which ...
0
votes
0answers
16 views

Correlations between target variables and other variables (binary)

I would like to calculate the correlation between the presence of uppercase, special punctuation, specific words, in texts labeled as fake/not fake. In order to do it, I created some columns in my ...
1
vote
0answers
9 views

Learn smoothly varying mean and variance of a variable over a 2d domain

For a problem which I am working on at the moment, I'm interested in learning how the mean and variance of some response variable y changes with two independent ...
0
votes
1answer
35 views

Understanding Confidence Interval

I am trying to understand the concept of Confidence Intervals. What is the meaning of point estimates and confidence intervals? What I understood is the point estimate in confidence interval is ...
2
votes
1answer
16 views

Assigning manually a score based on some statistics?

I am trying to assign manually scores to identify bad accounts (not in Stacks community!) based on some specific conditions, for example if they have account names containing only numbers and a ...
0
votes
0answers
17 views

How to generate synthetic data from training set for regression?

I have 41 training samples with 12 features and I am thinking about generating synthetic data that follows the distribution of my training set. All features are continuous variables. I found something ...
0
votes
0answers
4 views

Adding a feature increased amount of transaction however how do you prove it is not simply due to seasonal changes?

I've been always curious about similar topic however still do not fully understand it. Say in a car sharing company I am implementing promotion A from Sept ~ Dec. I want to know whether promotion had ...
1
vote
0answers
18 views

Generating synthetic data based off existing real data (in Python)

I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future ...
0
votes
0answers
4 views

Is there a way to compute micro-average for NDCG

I know how both micro and macro averages computed and there is a great discussion of the topic: Micro Average vs Macro average Performance in a Multiclass classification setting The macro-average for ...
0
votes
0answers
7 views

What is the explanation of discretized log logistic in the following code? Is there any reference regarding this piece of code?

I have taken the following piece of code from the OpenAI code of GitHub from the following link: https://github.com/openai/iaf/blob/master/tf_utils/distributions.py#L28 def discretized_logistic(mean, ...
0
votes
1answer
19 views

Pull Random Numbers from my Data (Python)

Let's imagine I have a series of numbers that represents cash flows into some account over the past 30 days in some time window. This data is non-normal but it does represent some distribution. I ...
1
vote
0answers
12 views

How much information is produced in our Society? [closed]

I was reading an interesting article about the growth of data in our society: https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2012.00584.x Does any of you know if there exists a more ...
0
votes
0answers
26 views

Bayesian Regression Model

I am new to Bayesian modeling. I am running Bayesian regression model in R using brm function from brms library, which is powered by STAN. I have a data with 10 million records. I took 10% sample out ...
1
vote
1answer
32 views

Hypothesis vs Hyperplane in Machine Learning

I am finding it hard to understand the clear difference between Hypothesis and Hyperplane. I know that Hypothesis is a candidate model that maps inputs to outputs after training . And , Hyperplane is ...
0
votes
1answer
36 views

Analysis of passed/not passed students by enrolled year

I am analysing some data from my University about successful students through years. The dataset looks like as follows ...
1
vote
1answer
15 views

predict_proba() on a continuos target made binary

I am building a Newtons gravitation equation model. It receives two mass and a radius and outputs what it generates from the formula. $F = G \frac{m1 · m2}{r^2}$ I want to transform it now to a binary ...
0
votes
1answer
26 views

Determine relationship between users and age?

I would like to understand how to find an association between users, spam and email's age. My dataset looks like as follows: ...
1
vote
0answers
19 views

What is the algorithm/logic top e-commerce companies use to show products in “What's new?” and “Bestsellers” sections on the website?

I am trying to solve this problem for a fashion e-commerce website that sells t-shirts, joggers etc. Till now I am only able to make some rule based logics only. It looks something like this The ...
0
votes
0answers
32 views

Odd results on bias-variance tradeoff assessment

I am running a bias-variance tradeoff assessment on ten regression models of increasing complexity (linear to x^10), but my results do not satisfy the MSE = Bias^2 + Variance + True Error relationship....
1
vote
1answer
31 views

What is the difference between maximum likelihood hypothesis and maximum a posteriori hypothesis?

I am a student and I am studying machine learning. I am focusing on the concept of Bayesian learning and I have studied the maximum likelihood hypothesis and the maximum a posteriori hypothesis. I ...
0
votes
3answers
23 views

Can samples of two different size be compared

I have two samples and the difference between their size is not too different like 100 and 90. I was thinking of comparing them using t-test. Is it fine to do this ? The sample contains quantitative ...
-1
votes
1answer
27 views

How we can identify the problem of Overfitting and underfitting and maintain bias?

Basically, I'm new to the data science field, and I'm getting a little bit of confusion about overfitting and underfitting. Are overfitting and underfitting is ...
0
votes
0answers
10 views

Deviation towards the higher number side

Let's say I have 10000 different observations. I want to calculate the mean deviation only towards the higher side of the observation. Do we have any term in statics for such a parameter?
0
votes
1answer
42 views

What is the best way to compare these small distributions?

I have one distribution of size 30. This are results (ROC-AUC for example) from training a neural network for 30 times in a row with the same hyperparameters but since they are randomly initialized ...
5
votes
1answer
103 views

Understand the equations of quantile regression forest (Meinshausen)?

I am trying to implement a quantile regression forest (https://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf). But, I have some difficulties to understand how the quantiles are ...
0
votes
0answers
6 views

What affects blurriness of images outputted by vanilla VAE?

I know that MSE between reconstruction (m(z)) and original image (X) affects blurriness, because it averages across pixel values. I also know that variance of the Gaussian distribution p(x|z) (p(x|z) =...
1
vote
1answer
25 views

Statistical test to establish that event B is caused by event A

I want to have a statistical test to establish that event B is caused by event A Here event B is customers that are supposed to leaved leave for competitor stay in our company event A is an attractive ...
2
votes
0answers
32 views

Plotting regression test errors

I am trying out an R exercise where I have to randomly split my data into train/test set 50 times. On each iteration, I use lm() to train polynomial models up to ...
0
votes
1answer
33 views

Estimate correlation in Python

I have a dataset with labels and usernames: ...
2
votes
1answer
33 views

How to account for reduced student capacity when calculating program retention?

I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer. I am ...
1
vote
0answers
22 views

Comparing different machine learning methods over multiple test datasets with different number of samples [closed]

Say, I have an image dataset (for example, imagenet) and I am training two image recognition models on it. I train a resnet with 10 layers 3 times on it (each time with different random weight ...
0
votes
0answers
21 views

What observations can you perceive from the following table?

I am preparing my thesis presentation and I want to give a clear understanding of the data that I have collected for the jury members. I understood the data very well, but I feel like I am lacking a ...
0
votes
0answers
16 views

How to get Optimum Lag with Data2 tends to catch up more on the upside than on the downside of data1

Hope you are doing well!! I have two time series, Data1 and Data2 from 01/01/2017 to 01/01/2020, I did a cross correlation with all the dates (from 01/01/2017 to 01/01/2020) to get the optimum lag, ...
0
votes
0answers
21 views

How to downsample a dataset with constraints in Python?

I have a dataset that has columns a, b, and others. Both a and ...
0
votes
0answers
23 views

Sampling methods for Text datasets (NLP)

I am working on two text datasets, one is having 68k text samples and other is having 100k text samples. I have encoded the text datasets into bert embedding. ...
1
vote
1answer
29 views

Poisson distribution, Standard deviation, fitting line

Let's say I have system A, B, C, and D. Each system contains 10000 numbers generated by Poisson distribution. The difference is the mean is different for different systems. I calculated the std dev ...
-1
votes
1answer
31 views

Equivalence of two separate data distribution

I am working in generative modelling. I have a real dataset of which I want to know the underlying distribution. For this I create a synthetic data generator which tries to mimic the real data. I ...
12
votes
3answers
1k views

Overfitting in Linear Regression

I'm just getting started with machine learning and I have trouble understanding how overfitting can happen in a linear regression model. Considering we use only 2 feature variables to train a model, ...
0
votes
1answer
37 views

Probability of an event [closed]

How should I go about solving the following problem? If you randomly type a 6-digit number on a note, what is the probability that you can see the same number if you flip your note upside down? How ...
2
votes
1answer
20 views

Interpreting confidence interval results for datasets

I have created a dataset automatically and wanted to clarify my interpretation of the amount of noise using the confidence interval. I selected a random sample and manually annotated the sample and ...
1
vote
2answers
26 views

What is the best way to calculate score based on input features?

Lets say we have certain products. We also have certain input features regarding these products like inventory, sales, price, cost etc. based on these features we want to score these parts. The score ...
0
votes
0answers
16 views

Identifying anomalies in spatial (latitude, longitude, time) data? [duplicate]

I'll start off by prefacing this with the fact that I'm not even sure if I'm asking the right question and what I'm really looking for is some guidance to dig in the right direction. I have a set of ...
0
votes
1answer
25 views

Alternating Least Squares

can anyone explain difference between Alternating Least Squares(ALS) and the recommendation systems? it will be helpful if you give me an example.
0
votes
0answers
22 views

Is there a certain threshold over which to accept or reject predictors based on correlation values with the target variable?

I have been working on the Titanic dataset. After some feature manipulation, I printed out the correlation values between my target variable Survived and all the ...

1
2 3 4 5
17