Skip to main content

Questions tagged [distribution]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
64 views

Using Maximum Mean Discrepancy (MMD) to Compare Kernel Density Estimates (KDEs)

I'm interested in comparing two Kernel Density Estimates (KDEs) and I've come across the Maximum Mean Discrepancy (MMD) metric ...
Adham Enaya's user avatar
0 votes
1 answer
51 views

Is Maximum Mean Discrepancy (MMD) suitable for comparing distributions with different sample sizes?

I'm working on a project where I need to compare the similarity of two probability distributions using MMD. However, the two datasets have different sample sizes. I've read that MMD can be affected by ...
Adham Enaya's user avatar
0 votes
0 answers
14 views

Two overlap gaussian mixtures come from one or two generations of data?

Reviewing for a ML exam, my teacher asked last year a philosophical question about gaussians and generations of data which I couldn’t find no good answer. Imagine that an oracle can say with 100% ...
Adrian Lara's user avatar
0 votes
1 answer
32 views

Binary Classification Dataset Distribution

I have a question regarding the distribution of my dataset. I have a binary classification task which I need to label. I now want to know how I should distribute the labels for optimal training / ...
Apatus's user avatar
  • 1
1 vote
2 answers
60 views

Is there a measure to compare features on the basis of normality

I have a dataset of cars and it has many features including 'acceleration’, ‘horsepower’, and ‘mpg'. I am supposed to check which of these features is the most similar to a normal distribution, so I ...
Tareq Ewaida's user avatar
0 votes
0 answers
25 views

What can I do when validation data and test data has different distribution in imbalance classification?

I am building classification model for bio (scRNA) data. Datasets in this field, for example, dataset A has 1, 2 classes, dataset B has 2, 3 classes kind of that. So I integrated datasets for training ...
containletters's user avatar
0 votes
0 answers
30 views

Bayesian Linear regression for non-gaussian distribution

Which distribution should be considered for non-gaussian distribution while building Bayesian Linear regression. I am trying to build this model using Pymc3 library but since I have just started to ...
Archaeolexicologist's user avatar
0 votes
0 answers
8 views

Grouped Time Restricted Demand Regression with value cap

so I am working on quite an interesting regression task that I haven't encountered before. Our company sells products (steel) in tons. We offer contracts where the customer orders a certain amount of ...
Martin Pichler's user avatar
0 votes
1 answer
359 views

What is a good metric for comparing a single value to a distribution?

I'd like to compare single values from one distribution to another distribution, effectively transforming the second distribution in such way, that its values reflect both datasets simultaneously, i.e....
Hayes's user avatar
  • 1
0 votes
0 answers
21 views

What distribution-shift datasets exist for ML?

What datasets there are which contain a (labelled) distribution shift or which distribution-shifted datasets share classes? Currently I have found some interesting results using CIFAR10 and the ...
Somniare's user avatar
0 votes
0 answers
50 views

Train neural network to predict multiple distributions

I aim to train a neural network to predict 2 distributions (10 quantiles, i.e. deciles) at 5 time points. So my y is of shape: ...
A_Murphy's user avatar
0 votes
0 answers
18 views

Using neural networks to learn total distributions from linearly transformed samples

I am trying to explore the use of neural network based models to replace a cumbersome challenge with a faster more approximate method. The problem involves learning a distribution termed the "...
user8188120's user avatar
0 votes
1 answer
61 views

For standard deviation's formula, why does division by sample size come before square rooting?

Why is this used to calculate a sample's standard deviation: $s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N(x_i-\bar{x})^2}$ and not something like: $s=\frac{1}{N-1}\sqrt{\sum_{i=1}^N(x_i-\bar{x})^2}$ I ...
Daniel Jiang's user avatar
0 votes
0 answers
63 views

Distribution check - different results using plt.hist and seaborn.displot

I have an apparently very simple task: checking the distribution of values contained in an array data. Only thing is, I get two completely different results by ...
sato's user avatar
  • 101
1 vote
0 answers
60 views

Understanding Syntactic divergence

I am trying to read a paper https://arxiv.org/abs/2004.14444 Section 6.1 of the paper describes Syntactic divergence. I have confusion regarding the distribution graphs and split of the dataset. If a ...
jon's user avatar
  • 11
3 votes
1 answer
2k views

What is the difference between Covariate Shift, Label Shift, Concept Shift, Concept Drift, and Prior Probability Shift?

As a beginner in MLOps, I was overwhelmed by some confusing definitions. As far as I understand, when we have a classifier or regressor with y = f(X) function: <...
Mohsen Mahmoodzadeh's user avatar
1 vote
0 answers
24 views

Strange behaviour around zero for predicted distributions from a deep learning regression model

Can somebody help make sense of these very odd distributions that I obtained from my trained deep learning regression model? The model was trained with either MAE or MSE loss, which is what the ...
Daan Scheepens's user avatar
0 votes
1 answer
94 views

What is the next after finding best distribution for my data?

I found the distribution of my data with "distfit" library for python. But what now? The best distribution that describes my data is "weibull" distribution. But I don't know what ...
Canovich's user avatar
0 votes
0 answers
51 views

Multiclass classification (gradient boosted trees) predictions distribution using softmax()

Let's consider a multiclass problem where the target is composed by 20% class 'A', 50% of class 'B' and 30% of class 'C'. The model is trained and then the class predictions are obtained via the ...
simon's user avatar
  • 133
0 votes
1 answer
197 views

Compute the pdf from pandas kde

I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...). I ...
Socorro's user avatar
  • 111
0 votes
1 answer
105 views

How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?

I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following: from this plot I see that ...
HelpNeederStudent's user avatar
0 votes
0 answers
27 views

Task Distribution

I would like to distribute tasks (going to companies and signing a deal) to sales representatives based on some condition like company location (is it near the representative's place) or profit of the ...
Joshua's user avatar
  • 1
1 vote
1 answer
48 views

name of log(n+1) plot

I am trying to plot a distribution of positive integers which contains a lot of variance. I opted to use the log of the y-values but that causes issues due to the inclusion of zeros. I though of ...
iHnR's user avatar
  • 113
0 votes
2 answers
2k views

How to plot categorical variables with a pie chart

I am concerned with a single column (fruit) from my df: ...
Tom Bomer's user avatar
1 vote
0 answers
14 views

How can PCA be distributed among workers?

I want to distribute the work from PCA among a set of workers. So let samples $x_1,... , x_d\in \mathbb{R}^n$ be samples. Then to find a dimension reduction subspace we need covariance matrix $cov(\...
A.Dumas's user avatar
  • 111
0 votes
0 answers
83 views

How does factoring probability distributions help?

I'm trying to make sense of machine learning and am reading Deep Learning by Goodfellow, Bengio and Courville. In section 3.14 they show an example of factoring a distribution then say "These ...
Alan's user avatar
  • 1
2 votes
1 answer
54 views

How to measure statistical similarity or discrepancy between a dataset and a distribution?

Is any way to measure statistical similarity or discrepancy between a dataset and a distribution? I have do some research, but find most of method are intended to describe discrepancy between data and ...
nick's user avatar
  • 21
0 votes
1 answer
27 views

Is it possible to find an internal event in a Classification between two classes?

I am very new to the machine learning area. So my question might be trivial  I have two classes $U, V$ of binary vectors. In the training phase,  I use $u_1,\ldots, u_{1000}$ from $U$ class and $v_1, \...
Sanu's user avatar
  • 1
0 votes
0 answers
80 views

Function for KDE-style distribution generation for sampling

I have some points in pytorch and I would like to sample from a distribution that resembles these points. I noticed that the seaborn kde plots seem to draw out/define a distribution graphically and I ...
nighthawk's user avatar
1 vote
1 answer
26 views

What thought processes do people use for generating priors on a variable's probability distribution?

Example Consider a block of text with a variety of sentence types within it, of which there are 7. Within a text these sentences will be more or less likely to appear, dependent on where in the text ...
quanty's user avatar
  • 133
1 vote
1 answer
87 views

Handling gaps in regression model

I'm facing a regression problem where I'm supposed to predict the delay of some trains. There's some peculiar particularity, however: a train is not considered delayed until it has more than 10 mins ...
nprime496's user avatar
0 votes
2 answers
3k views

Comparing distributions Python

I have a larger dataset (random variable) 'x' containing values approximating a Gaussian distribution. From 'x', a much smaller random variable 'y' is sampled without replacement. I want to compare ...
Arun's user avatar
  • 121
1 vote
0 answers
19 views

Probability notation q(y) and q(Y) and its implication to vector functions

The function in question is (from Appendix B, Proof of proposition 2.1 from Posterior Regularization for Structured Latent Variable Models): $$q(\textbf{Z}) = \frac{p_{\theta}(\textbf{Z}|\textbf{X})...
Martin G's user avatar
  • 131
2 votes
2 answers
1k views

Does t-test require Standard Deviation of sample for calculation

Might be a novice question, but the main difference between a t-test and z-test, I was able to understand, is that the z-test calculation requires the SD value of the population where as in a t-test, ...
Chris_007's user avatar
  • 193
-1 votes
1 answer
55 views

Normalizing Distributions for features in predicting

Should features used for predictions be normalized if they are highly skewed. Or should I only normalize target feature that is supposed to be predicted?
Zexxxx's user avatar
  • 33
0 votes
0 answers
24 views

How to do ? serious skewed right transformations normally distribution

My target is doing two-way Anova. But Here is some problem. This is my data like I try log(X). Looks like a normal distribution. But try Kolmogorov-Smirnov prove is not. Can anyone recommend another ...
ElenaWu's user avatar
4 votes
2 answers
683 views

Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
Student's user avatar
  • 411
1 vote
0 answers
23 views

Is there a way to create a continuous distribution graph from discrete percentiles?

I have an underlying distribution that I know is 1-tailed (Packet Response time), but the service that runs the benchmark is opaque to me. It only returns a percentile table up to some number of nines ...
knowads's user avatar
  • 11
0 votes
1 answer
354 views

Distribution of text data

How can I identify whether the training data and test data came from same distribution or not? I tried with TFIDF and cosine similarity ...
SS Varshini's user avatar
2 votes
0 answers
18 views

Measuring the distance between data points based on mutual linkages

How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages? I don't know the technical term for that, so here is a fictitious example from scientific ...
anpami's user avatar
  • 123
1 vote
1 answer
217 views

How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
Bazoya's user avatar
  • 31
1 vote
2 answers
566 views

Distribution Shift vs Transfer Learning

Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem [1] ...
Carlos Mougan's user avatar
1 vote
0 answers
2k views

How to generate a positive skewed array?

As the title suggests, I would like to specify the task that I really need help with: Given three possible distributions of an array: I know that given a skewed dataset (the left and the right), then ...
Miku's user avatar
  • 11
0 votes
0 answers
153 views

A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
Matt's user avatar
  • 821
1 vote
1 answer
43 views

Check if distribution per week is the same

I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week. Initial dataset Customer Date Sales Alpha 2019-02-23 527 ...
Ismail's user avatar
  • 111
1 vote
0 answers
21 views

Criteria for assessing difficulty of a question

I have a list of questions and how many times they have been answered correctly and incorrectly. Based on this, I applied the formula: ...
n.mathfreak's user avatar
1 vote
0 answers
31 views

Testing a Binary Classifier

I have been training a binary multilayer perceptron on a database made out of roughly 3600 0 values, and 4 1 values. Afterwards, I'm testing the MLP on a test set made out of 7 0 values and 7 1 ...
Joni Joni -al's user avatar
1 vote
0 answers
74 views

Estimating the (normal?) distribution of the grayscale values of a region in camera footage

The problem: Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
John's user avatar
  • 111
0 votes
1 answer
93 views

Non IID variables and SVM Classifier

I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
Aditya Kulkarni's user avatar
1 vote
2 answers
2k views

Data Analytics how to read ECDF graph

Hi there, My question is about how to read ECDF graphs. I am still quite unsure what the jumps / zig-zags in the graph mean and what is happening when there is a horizontal line and so on. I would be ...
Yavuz Bozkurt's user avatar