# Questions tagged [distribution]

The tag has no usage guidance.

165 questions
Filter by
Sorted by
Tagged with
47 views

### Fit a bimodal histogram with a mixture Model that is not Gaussian

I am attempting to fit with Python a histogram which presents a bimodal distribution. For now, I have tested to use the Gaussian Mixture Model (GMM) from Scikit-Learn, but I want to try with different ...
1 vote
41 views

### For standard deviation's formula, why does division by sample size come before square rooting?

Why is this used to calculate a sample's standard deviation: $s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N(x_i-\bar{x})^2}$ and not something like: $s=\frac{1}{N-1}\sqrt{\sum_{i=1}^N(x_i-\bar{x})^2}$ I ...
10 views

### Normalizing flows with truth values?

I am interested in using normalizing flows to map to a certain distribution of points, but I want to make sure that the distribution is not just of the correct general shape, but also that specific ...
18 views

### Distribution check - different results using plt.hist and seaborn.displot

I have an apparently very simple task: checking the distribution of values contained in an array data. Only thing is, I get two completely different results by ...
1 vote
28 views

### Understanding Syntactic divergence

I am trying to read a paper https://arxiv.org/abs/2004.14444 Section 6.1 of the paper describes Syntactic divergence. I have confusion regarding the distribution graphs and split of the dataset. If a ...
214 views

### What is the difference between Covariate Shift, Label Shift, Concept Shift, Concept Drift, and Prior Probability Shift?

As a beginner in MLOps, I was overwhelmed by some confusing definitions. As far as I understand, when we have a classifier or regressor with y = f(X) function: <...
1 vote
8 views

### Strange behaviour around zero for predicted distributions from a deep learning regression model

Can somebody help make sense of these very odd distributions that I obtained from my trained deep learning regression model? The model was trained with either MAE or MSE loss, which is what the ...
52 views

### What is the next after finding best distribution for my data?

I found the distribution of my data with "distfit" library for python. But what now? The best distribution that describes my data is "weibull" distribution. But I don't know what ...
15 views

### How to calculate the Syntactic divergence in Question Answering dataset?

I am trying to calculate the Syntactic divergence for my private Question Answering dataset, but I couldn't find any good implementation or explanation on how to do it in Python. While Searching, I ...
20 views

### What method of imputation would you choose for a distribution like this

From the following image, it is obvious that while the values start to converge after a while, close to zero they are unstable. What method of imputation would you choose for a distribution like this, ...
41 views

### Multiclass classification (gradient boosted trees) predictions distribution using softmax()

Let's consider a multiclass problem where the target is composed by 20% class 'A', 50% of class 'B' and 30% of class 'C'. The model is trained and then the class predictions are obtained via the ...
68 views

### Compute the pdf from pandas kde

I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...). I ...
57 views

### How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?

I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following: from this plot I see that ...
17 views

I would like to distribute tasks (going to companies and signing a deal) to sales representatives based on some condition like company location (is it near the representative's place) or profit of the ...
1 vote
39 views

### name of log(n+1) plot

I am trying to plot a distribution of positive integers which contains a lot of variance. I opted to use the log of the y-values but that causes issues due to the inclusion of zeros. I though of ...
984 views

### How to plot categorical variables with a pie chart

I am concerned with a single column (fruit) from my df: ...
10 views

### Split 2 zipfian distributions

Suppose I have 2 or more sequences distributed by Zipf's law merged together. It would look like this on log-log graph after sorting: or like this: What would be the best way to split them? I cannot ...
1 vote
14 views

50 views

### Function for KDE-style distribution generation for sampling

I have some points in pytorch and I would like to sample from a distribution that resembles these points. I noticed that the seaborn kde plots seem to draw out/define a distribution graphically and I ...
1 vote
23 views

### What thought processes do people use for generating priors on a variable's probability distribution?

Example Consider a block of text with a variety of sentence types within it, of which there are 7. Within a text these sentences will be more or less likely to appear, dependent on where in the text ...
1 vote
43 views

### Handling gaps in regression model

I'm facing a regression problem where I'm supposed to predict the delay of some trains. There's some peculiar particularity, however: a train is not considered delayed until it has more than 10 mins ...
1k views

### Comparing distributions Python

I have a larger dataset (random variable) 'x' containing values approximating a Gaussian distribution. From 'x', a much smaller random variable 'y' is sampled without replacement. I want to compare ...
1 vote
17 views

### Probability notation q(y) and q(Y) and its implication to vector functions

The function in question is (from Appendix B, Proof of proposition 2.1 from Posterior Regularization for Structured Latent Variable Models): q(\textbf{Z}) = \frac{p_{\theta}(\textbf{Z}|\textbf{X})...
313 views

### Does t-test require Standard Deviation of sample for calculation

Might be a novice question, but the main difference between a t-test and z-test, I was able to understand, is that the z-test calculation requires the SD value of the population where as in a t-test, ...
38 views

### Normalizing Distributions for features in predicting

Should features used for predictions be normalized if they are highly skewed. Or should I only normalize target feature that is supposed to be predicted?
23 views

### How to do ? serious skewed right transformations normally distribution

My target is doing two-way Anova. But Here is some problem. This is my data like I try log(X). Looks like a normal distribution. But try Kolmogorov-Smirnov prove is not. Can anyone recommend another ...
329 views

### Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
1 vote
16 views

### Is there a way to create a continuous distribution graph from discrete percentiles?

I have an underlying distribution that I know is 1-tailed (Packet Response time), but the service that runs the benchmark is opaque to me. It only returns a percentile table up to some number of nines ...
175 views

### Distribution of text data

How can I identify whether the training data and test data came from same distribution or not? I tried with TFIDF and cosine similarity ...
16 views

### Measuring the distance between data points based on mutual linkages

How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages? I don't know the technical term for that, so here is a fictitious example from scientific ...
1 vote
178 views

### How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
1 vote
285 views

### Distribution Shift vs Transfer Learning

Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem  ...
1 vote
1k views

### How to generate a positive skewed array?

As the title suggests, I would like to specify the task that I really need help with: Given three possible distributions of an array: I know that given a skewed dataset (the left and the right), then ...
112 views

### A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
1 vote
43 views

### Check if distribution per week is the same

I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week. Initial dataset Customer Date Sales Alpha 2019-02-23 527 ...
1 vote
18 views

### Criteria for assessing difficulty of a question

I have a list of questions and how many times they have been answered correctly and incorrectly. Based on this, I applied the formula: ...
1 vote
30 views

### Testing a Binary Classifier

I have been training a binary multilayer perceptron on a database made out of roughly 3600 0 values, and 4 1 values. Afterwards, I'm testing the MLP on a test set made out of 7 0 values and 7 1 ...
1 vote
74 views

### Estimating the (normal?) distribution of the grayscale values of a region in camera footage

The problem: Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
62 views

### Non IID variables and SVM Classifier

I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
789 views

### Data Analytics how to read ECDF graph

Hi there, My question is about how to read ECDF graphs. I am still quite unsure what the jumps / zig-zags in the graph mean and what is happening when there is a horizontal line and so on. I would be ...
518 views

### How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
1 vote
415 views

### Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
1 vote