Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [distribution]

The tag has no usage guidance.

0
votes
0answers
6 views

How to smoothen array of samples?

I am implementing a Genetic Algorithm library. However, on Roulette Wheel Selection algorithm sometimes one or two chromosomes dominate he roulette wheel with their relatively large fitness scores. I ...
0
votes
1answer
14 views

KL divergence in VAE

If I understand correctly KL-divergence is relative entropy of two distributions. To calculate KL divergence of two distributions, you would need two vectors of random variables. What I do not ...
0
votes
0answers
9 views

VAE: Calculating reconstruction loss when the true posterior is continuous

In Python / Tensorflow, I am currently struggling with how to compute the VAE reconstruction loss $ E_{q_\phi(z|x^{(i)})} [log p_{\theta} (x^{(i)}|z)] $ (which is equivalent to the negative cross ...
1
vote
0answers
13 views

How to check the similarity between two transition matrixes

I have two transition matrixes in which the probabilities for the transition between each from state to each to states are ...
0
votes
0answers
14 views

Titanic Disaster Problem in Kaggle: distribution of numerical feature values across the samples

I'm still a newbie in Data Science and I found the Titanic question and solution here in Kaggle. I've been trying to understand the solution yet I still can't grasp why he used those percentiles for ...
4
votes
2answers
107 views

Standard Deviation for Z-scores

I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of: $\sigma = \sqrt{p * n * (1-p)}$ My data is ...
1
vote
0answers
12 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
0
votes
1answer
30 views

Generating a set of different scenarios based on some initial observations

I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for ...
0
votes
2answers
38 views

Maximum likelihood estimation vs calculating distribution parameters “manually”

I'm sorry for asking probably elementary question, but I cannot understand how estimating probability distribution parameters using maximum likelihood estimation method differs from calculating these ...
0
votes
1answer
112 views

What Does the Normalization Factor Mean in the AdaBoost Algorithm?

I am studying the AdaBoost algorithm. The update rule for a weak hypothesis is: $Dt+1(i) = Dt(i)exp(−αtyiht(xi))/zt $ where $zt$ is a normalization factor chosen so that $Dt+1$ is a distribution. ...
1
vote
1answer
23 views

Calculate whether datapoints are part of a larger distribution

I have some normally distributed variables (~800) and some variables that are in some way special (~30). I need to find out whether the special ones can be considered normal members of the ...
0
votes
0answers
25 views

Why use parametric test at all if non parametric tests are 'less strict'

I have read from several sources, even in my undergrad courses, that parametric tests require the data to have a certain distribution, for instance normal, whilst non-parametric don't. I have ...
1
vote
1answer
59 views

How often do we see normally distributed data

I am having difficulty in exactly understanding several statistical tests, such as the t-test and ANOVA test. These tests require that the data we use be normally distributed. However, whilst ...
1
vote
0answers
13 views

How to get maintenance interval from maintenance outcomes?

I have a machine, which needs maintenance. Every time the technician visits the machine, there are four possible outcomes: a) The machine is broken, b) The machine is still running, but it's high ...
0
votes
0answers
6 views

Measuring minority class frequency (with confidence interval) with calibration data

I have a large amount of unlabeled data, a binary classifier that outputs a score (uncalibrated) and a small amount of labeled data that was used to create a calibration plot. When I run my ...
0
votes
2answers
34 views

Binomial family in logistic regression

I was asked in an interview why do we use the binomial distribution in logistic regression and how is it related to the class that we are predicting? Could anyone explain, without any mathematical ...
0
votes
0answers
25 views

Computing probability distribution given distribution of 2 variables

I have a matrix that represents the distribution of 2 variables. As an example the 1 st row (after the header) in the table below represents percentage of values falling in different percentile ...
1
vote
0answers
16 views

On Noise Contrastive Estimation, replace noise distribution with difficult examples

In noise contrastive estimation, we learn a binary classifier to classify noise from the true distribution. The problem I'm trying to solve has discrete input variables yet it's intractable to sum ...
2
votes
0answers
90 views

Wasserstein distance between Gaussian and the empirical distribution

Wasserstein distance between two gaussians has a closed form solution. Does the same hold for the distance between a Gaussian with a fixed variance(say 1) and the empirical data distribution? ...
0
votes
0answers
43 views

How to better visualizing Earth Mover Distance in relation with generative adversarial networks?

After reading the original EMD paper from 1998 I am having a hard time trying to visualize the connections between their dirt pile example to generative adversarial networks. Everything is kind of ...
0
votes
1answer
26 views
1
vote
1answer
58 views

Does classification of a balanced data-set lead to any problem?

So I came across a bioinformatics paper, where I found a line which says: One potential problem with using a training set with equal numbers of positive and negative examples in cross-validation ...
2
votes
2answers
76 views

can I say that my variable is “approximately” normally distributed?

I am studying a variable on 50000 observations, I have applied a cox box transformation to make it normal, but even with transformation the kolmogorov smirnof and the anderson darling test tells that ...
0
votes
1answer
16 views

Feature has a pattern in relation to class but does not enhance classifier to predict class

I have a feature x that when I plot it again my class variable y it shows some sort of pattern, i.e. it is obvious that x has a relation with y, but when I add the x to my logistic model it reduce the ...
0
votes
0answers
25 views

Importance Sampling for Minibatches

I have a question regarding to importance sampling of subsets. Given a Dataset of 943 by 1682, I need to create subsets of the data and provide each subset with a weight v_i. Hence, my first question ...
1
vote
2answers
40 views

Algorithms that would benefit from variable transformations?

1- Which algorithms would benefit from data that has been transformed, so that distributions of continuous variables resemble that of a normal distribution ? 2- What would be the benefits of ...
0
votes
0answers
4 views

Quantiles for queuing processes with exponential arrivals and Pareto departures

I am generating the data from process $X_{n}= X_{n-1} + U_n - W_n $ where $ U_n= [U_{n1}, U_{n2}]$ is bivariate exponential process with correlation and $W_n= [W_{n1}, W_{n2}]$ is bivariate Pareto ...
1
vote
0answers
13 views

Number of events estimation

I have three different histograms (Impact parameter distributions) corresponding to three groups of the same particle with different properties. However, the three distribution have more or less the ...
1
vote
2answers
79 views

Boxplots or violinplots?

This is quite a general question, perhaps somewhat opinion-based. In most papers people use boxplots to visualize a certain distribution, yet violinplots are able to give more information. ...
0
votes
1answer
50 views

Constructing graph of crypto financial instruments 2016-2017

Set of financial instruments represent the set of vertices of the graph. For any pair of vertices $i$ and $j$, an edge connecting them is added to the graph if the corresponding correlation ...
2
votes
1answer
34 views

Stratified Sampling Variable Choice

I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set. Is ...
2
votes
2answers
590 views

Working with Data which is not Normal/Gaussian

What happens if my data/feature is not normal? Can I still use machine learning algorithms to utilize such data for predictions? I noticed in many data sciences courses, there is always a strong ...
0
votes
2answers
59 views

Detecting Different Distributions in data [closed]

Supposing I have a dataset that I assume that have instances generated by two different distributions, is it possible to separate these instances based on the underlying generating distribution? ...
2
votes
2answers
127 views

Machine learning learn to work well on future data distribution?

This is based on my limited machine learning scope and experience, so correct me if I'm wrong. Many of the currently used machine learning models (SVMs, boosted trees, DNNs) work under the assumption ...
9
votes
2answers
595 views

Which outlier detection can detect these outliers?

I have a vector and want to detect outliers in it. The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal. ...
0
votes
1answer
36 views

Estimate the normal distribution of the mean of a normal distribution given a set of samples?

Let's say there is a distribution, call it D, for which I don't know details (i.e. mean and variance) but can assume that it's a normal distribution. I now have N samples from D. I cannot take more ...
2
votes
0answers
239 views

Maximum likelihood Estimation of three-parameter log-logistic distribution in R

I am trying to estimate parameters from a three-parameter Log-logistic distribution in R. I have a code as follows: ...
1
vote
2answers
608 views

Does SQL Server support the Poisson distribution?

I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson ...
-1
votes
1answer
97 views

Gamma random variable , need to find the approximate 90th percentile of X? [closed]

A colleague defines a random variable $X = \frac{Z}{Y^2}$, where $Z$ is a known normal random variable, $Y$ is a known gamma random variable, and $Z$ and $Y$ are independent of each other. You are ...
1
vote
0answers
432 views

Generalization error for simple linear regression

Lets say we have a training data and we have estimated a fit for a model of square ft of living area vs price of houses. Suppose we know the probability distribution of sq ft and for a fixed sq ft we ...
0
votes
0answers
42 views

Is there any known distribution I can resort to?

I face a dataset which describe out of each quarterly total loan amount of a bank, how much (dollar amount) was charged off. For example, say 2015 Qrt 1, the total loan amount is 10,000 USD, by some ...
2
votes
1answer
320 views

Methods / Algorithms for rank scales based on cumulative scoring

Say you have an organization that requires employees to participate in a Q&A site similar to StackOverflow - questions and answers are voted upon, selected answers get extra points, certain ...
5
votes
1answer
4k views

Plotting different values in pandas histogram with different colors

I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0,1,2). In order to check the distribution of values in each column, I used <...
-1
votes
3answers
429 views

How to check if a data is in gaussian distribution in R or excel?

I know about the fitdist() function from the fitdistrplus package in R, however, I am not able to use it to predict a gaussian ...
3
votes
1answer
4k views

Transform a skewed distribution into a Gaussian distribution

I have a skewed distribution that looks like this: How can I transform it to a Gaussian distribution? The values represent ranks, so modifying the values does not cause information loss as long as ...
3
votes
1answer
587 views

How to estimate the mutual information numerically?

Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ? ...
2
votes
1answer
134 views

Shifted feature distribution across different datasets

I am trying to validate a classifier using two different training and testing datasets. The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. ...