Questions tagged [distribution]
The distribution tag has no usage guidance.
0
votes
0answers
7 views
How does t-sne scale work? Is it possible to compare scales of two t-sne plots?
I have used t-sne for two convolutional layers of binary classifier (VGG16 like CNN). I want to compare them, but I am not sure if that is okay as the scales of these two outputs from t-SNE plots can ...
0
votes
0answers
7 views
Normalization in SVM classifier
I am trying to normalize my features for a classification model with 3 class outputs. There are two kinds of features. First is medical test results and second is patient information such as age. The ...
0
votes
0answers
52 views
Normalization and probability density function
I am wondering is it reasonable to calculate the probability density function ( or calculate some measures of the probability distribution such as skewness) of data after standardizing the data (or ...
5
votes
2answers
70 views
how to check the distribution of the training set and testing set are similar
I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
0
votes
0answers
9 views
create joint prob distribution or empirical relation for two variables
There are two variables, X1 and X2. The experimental study shows that they are highly correlated. Are there any reliable ways to create an empirical mapping(or equation) between X1 and X2. Assuming ...
0
votes
1answer
27 views
Distribution of error values in linear regression vs logistic regression
Why do error values in linear regression have to be normally distributed and why not in logistic regression?
2
votes
0answers
18 views
How to create a prediction interval with the fact that the residuals follow a specific distribution (in python)
I am looking at a software development pipeline where I am predicting the lead time of different products flowing through the pipeline.
After applying a boxcox transformation on the lead time (...
2
votes
1answer
47 views
Normal distribution instead of Logistic distribution for classification
Logistic regression, based on the logistic function $\sigma(x) =
\frac{1}{1 + \exp(-x)}$, can be seen as a hypothesis testing problem. Where the reference distribution is the standard Logistic ...
0
votes
0answers
13 views
Is DQN independent from type of input distribution?Why?
Can we say:
Since DQN is online learning, it is independent of type of input distribution?
2
votes
2answers
75 views
How to visualize change in a distribution with a few outliers that account for a very large percent of the total?
I'm working on an edtech product where some of our traffic lands on webpages about textbooks.
Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "...
0
votes
1answer
74 views
Clustering by Distributions of Groups of observations
I have data as shown below. Like the groups shown A,B,C.. there are roughly 5400 groups of data and 15 observations for every group (I sampled 6-7 for each group and pasted them here).
...
1
vote
0answers
9 views
How to elegantly caclulate probability distribution parameters for a particular random variable given some observed data?
I am using bnlearn package and R to learn Bayesian Network structure and also fit it using Maximum Likelihood estimation(MLE). bn.fit uses MLE to learn[as i understand] a generalized probability ...
4
votes
1answer
204 views
How can I plot/display a dataset or an image distribution?
I want to view a specific image or a dataset's distribution, and see if they are different.
Does simply writing something like :
...
1
vote
0answers
46 views
Two-sample K–S test Kolmogorov–Smirnov test in Python
For my evaluation, I have three different time-series data of the following format with different characterstics where the first column is timestamp and the second column is the value.
...
1
vote
0answers
22 views
Probability Distribution of the Process Duration Time based on the Process Status Measurements
I have data of execution of several processes. Each execution is identified by the process_id.
The duration of the execution is measured by external agents, testing ...
3
votes
3answers
52 views
How to predict whether or not a customer will renew
I have a dataset of customer contracts that specify a start date and if applicable an end date. Each month a customer is up for renewal. Below is an example of how the data is organized in excel:
<...
0
votes
0answers
21 views
Checking skewness of variables
I follow a tutorial for the current dataset I'm working on. In this tutorial the target variable is check for skewness, and corrected het for by by taking the log transformation. Hereafter, skewness ...
0
votes
0answers
11 views
Need Weibull and Gamma Regression Data
I am working on regression with non-normal responses. Now i need a data set(s) in which responses follow Weibull and Gamma distribution, with some reference. kindly data may contain at least 15 ...
0
votes
0answers
10 views
Should we invest more in exploration (as opposed to exploitation) when the distribution is fat-tailed in contrast to a bell-curve?
Not sure if this falls under data science but here it goes:
In the exploration/exploitation trade-off (deliberation vs commitment to one choice), should we invest more in exploration when the ...
0
votes
1answer
212 views
KL divergence in VAE
If I understand correctly KL-divergence is relative entropy of two distributions. To calculate KL divergence of two distributions, you would need two vectors of random variables.
What I do not ...
1
vote
0answers
27 views
How to check the similarity between two transition matrixes
I have two transition matrixes in which the probabilities for the transition between each from state to each to states are ...
4
votes
2answers
112 views
Standard Deviation for Z-scores
I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of:
$\sigma = \sqrt{p * n * (1-p)}$
My data is ...
1
vote
0answers
13 views
How to optimize the separation of two distributions from binary classfication
Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
0
votes
1answer
35 views
Generating a set of different scenarios based on some initial observations
I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for ...
0
votes
2answers
68 views
Maximum likelihood estimation vs calculating distribution parameters “manually”
I'm sorry for asking probably elementary question, but I cannot understand how estimating probability distribution parameters using maximum likelihood estimation method differs from calculating these ...
0
votes
1answer
997 views
What Does the Normalization Factor Mean in the AdaBoost Algorithm?
I am studying the AdaBoost algorithm. The update rule for a weak hypothesis is:
$Dt+1(i) = Dt(i)exp(−αtyiht(xi))/zt $
where $zt$ is a normalization factor chosen so that $Dt+1$ is a distribution.
...
1
vote
1answer
23 views
Calculate whether datapoints are part of a larger distribution
I have some normally distributed variables (~800) and some variables that are in some way special (~30). I need to find out whether the special ones can be considered normal members of the ...
1
vote
1answer
61 views
How often do we see normally distributed data
I am having difficulty in exactly understanding several statistical tests, such as the t-test and ANOVA test. These tests require that the data we use be normally distributed.
However, whilst ...
1
vote
0answers
17 views
How to get maintenance interval from maintenance outcomes?
I have a machine, which needs maintenance.
Every time the technician visits the machine, there are four possible outcomes:
a) The machine is broken, b) The machine is still running, but it's high ...
0
votes
2answers
52 views
Binomial family in logistic regression
I was asked in an interview why do we use the binomial distribution in logistic regression and how is it related to the class that we are predicting?
Could anyone explain, without any mathematical ...
1
vote
0answers
26 views
On Noise Contrastive Estimation, replace noise distribution with difficult examples
In noise contrastive estimation, we learn a binary classifier to classify noise from the true distribution.
The problem I'm trying to solve has discrete input variables yet it's intractable to sum ...
2
votes
0answers
111 views
Wasserstein distance between Gaussian and the empirical distribution
Wasserstein distance between two gaussians has a closed form solution. Does the same hold for the distance between a Gaussian with a fixed variance(say 1) and the empirical data distribution?
...
0
votes
1answer
31 views
1
vote
1answer
103 views
Does classification of a balanced data-set lead to any problem?
So I came across a bioinformatics paper, where I found a line which says:
One potential problem with using a training set with
equal numbers of positive and negative examples in cross-validation
...
2
votes
2answers
147 views
can I say that my variable is “approximately” normally distributed?
I am studying a variable on 50000 observations, I have applied a cox box transformation to make it normal, but even with transformation the kolmogorov smirnof and the anderson darling test tells that ...
0
votes
1answer
17 views
Feature has a pattern in relation to class but does not enhance classifier to predict class
I have a feature x that when I plot it again my class variable y it shows some sort of pattern, i.e. it is obvious that x has a relation with y, but when I add the x to my logistic model it reduce the ...
1
vote
2answers
41 views
Algorithms that would benefit from variable transformations?
1- Which algorithms would benefit from data that has been transformed, so that distributions of continuous variables resemble that of a normal distribution ?
2- What would be the benefits of ...
1
vote
0answers
14 views
Number of events estimation
I have three different histograms (Impact parameter distributions) corresponding to three groups of the same particle with different properties. However, the three distribution have more or less the ...
1
vote
2answers
142 views
Boxplots or violinplots?
This is quite a general question, perhaps somewhat opinion-based.
In most papers people use boxplots to visualize a certain distribution, yet violinplots are able to give more information. ...
0
votes
1answer
53 views
Constructing graph of crypto financial instruments 2016-2017
Set of financial instruments represent the set of vertices of the graph. For any pair of vertices $i$ and $j$, an edge connecting them is added to the graph if the corresponding correlation ...
2
votes
1answer
55 views
Stratified Sampling Variable Choice
I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set.
Is ...
3
votes
2answers
1k views
Working with Data which is not Normal/Gaussian
What happens if my data/feature is not normal? Can I still use machine learning algorithms to utilize such data for predictions?
I noticed in many data sciences courses, there is always a strong ...
0
votes
2answers
61 views
Detecting Different Distributions in data [closed]
Supposing I have a dataset that I assume that have instances generated by two different distributions, is it possible to separate these instances based on the underlying generating distribution?
...
2
votes
2answers
147 views
Machine learning learn to work well on future data distribution?
This is based on my limited machine learning scope and experience, so correct me if I'm wrong. Many of the currently used machine learning models (SVMs, boosted trees, DNNs) work under the assumption ...
8
votes
2answers
671 views
Which outlier detection can detect these outliers?
I have a vector and want to detect outliers in it.
The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal.
...
0
votes
1answer
36 views
Estimate the normal distribution of the mean of a normal distribution given a set of samples?
Let's say there is a distribution, call it D, for which I don't know details (i.e. mean and variance) but can assume that it's a normal distribution. I now have N samples from D. I cannot take more ...
1
vote
0answers
294 views
Maximum likelihood Estimation of three-parameter log-logistic distribution in R
I am trying to estimate parameters from a three-parameter Log-logistic distribution in R. I have a code as follows:
...
1
vote
2answers
747 views
Does SQL Server support the Poisson distribution?
I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson ...
-1
votes
1answer
117 views
Gamma random variable , need to find the approximate 90th percentile of X? [closed]
A colleague defines a random variable $X = \frac{Z}{Y^2}$, where $Z$ is a known normal random variable, $Y$ is a known gamma random variable, and $Z$ and $Y$ are independent of each other.
You are ...
1
vote
0answers
553 views
Generalization error for simple linear regression
Lets say we have a training data and we have estimated a fit for a model of square ft of living area vs price of houses. Suppose we know the probability distribution of sq ft and for a fixed sq ft we ...
