Questions tagged [distribution]
The distribution tag has no usage guidance.
169 questions
0
votes
1
answer
45
views
How does $\chi^2$ feature selection work?
I can't find the information how $\chi^2$ are used to select numerical features for a model.
...
- 301
0
votes
1
answer
418
views
Can continuous dataset have negative values? [closed]
As in the description, I want to choose continuos dataset with temperatures and the minimun value is -40 and maximum is 51. Is that OK? Can we have negative values in continuous distribution?
- 131
2
votes
0
answers
238
views
Use of PYMC distribution.dist()
In pymc3 documentation, it specifies that the .dist() member for distribution allows the distribution to be used without a model for sampling and use of the logp functions e.g.
...
- 71
3
votes
2
answers
17k
views
When to use t-distribution instead of normal distribution?
According to the Student's t-distribution wiki article the t-distribution is used instead of the Normal distribution "when estimating the mean of a normally distributed population in situations where ...
- 203
6
votes
1
answer
4k
views
Why do train, test, validation datasets need to have the same distribution?
I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution.
Can ...
- 161
2
votes
1
answer
138
views
Co-joining multi-peak histograms
I am analysing a bunch of data files which represent responsiveness of cells to addition of a drug. If a drug is not added, cell responds normally, if it is added, it shows abnormal patterns:
, .
We ...
0
votes
1
answer
161
views
Predicted and true values distributions comparison
Is this alarming when a distribution of predicted values differs from a distribution of true values? I use xgbregressor and get the following plots
Usage of boxcox doesn't improve the case.
My data ...
- 301
2
votes
0
answers
816
views
difference between normal skewed distribution and skewed distribution [closed]
From what I've read normal skewed is a distribution that has all the properties of normal distribution and is skewed:
according to this resource:https://www.statisticshowto.datasciencecentral.com/...
- 479
2
votes
1
answer
118
views
How to combine data having similar distribution?
I have a collection of time series data with data points of around 2 years of daily data. I am thinking of a way to increase the number of data points in it so that the neural network gets a better ...
- 177
4
votes
2
answers
61
views
Is it possible to train probabilistic model to return several distributions?
I have nonlinear data of function y(x), which is let's say parabolic. At some points of x there are several y's (look at the picture).
Is it possible to train a probabilistic model to return several ...
- 141
2
votes
2
answers
968
views
Fitting a distribution to data
I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation ...
- 123
2
votes
3
answers
206
views
How this visualisation was made?
I really like how this visualization represents the survey participants.
Is any tool for that? (Or R/python library?)
9
votes
3
answers
6k
views
xgboost: Is there a way to perform regression on rates/percentages data?
I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific ...
- 300
0
votes
1
answer
888
views
How to generate 12 independent random weights which all add up to one
I'm using Palisade's @Risk software with a triangular distribution to generate 12 random weights which must add up to one, but I get a lot of negative numbers. Is there a straightforward way to set ...
- 103
0
votes
1
answer
1k
views
CUDA 8.0 is compatible with my GeForce GTX 670M Wikipedia says, but TensorFlow rises an error: GTX 670M's Compute Capability is < 3.0
According to Wikipedia, the GeForce GTX 670M has a Compute Capability of 2.1 (and a Fermi micro-architecture), which is confirmed by TensorFlow (I can read "2.1" in the error it rises).
...
4
votes
2
answers
3k
views
Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?
I'm looking at telecom customers data. Two of the variables I'm looking at currently are:
Monthly Charges - The total amount charged to the customer monthly.
Is Senior Citizen - Whether the customer ...
- 43
1
vote
2
answers
214
views
Effect of skewness in data
I am preparing classification model. Many of numeric variables are positives skewed. Should I change a distribution of variables to be more Gaussian?
- 19
0
votes
2
answers
73
views
How do I fix mis-rendered matplotlib?
How do I correct my data or format it so that it is presentable, and fix my graphs?
Dataset is 345551 rows × 7 columns.
I am using numpy, pandas, seaborn and matplot lib.
It seems that my pricing ...
1
vote
1
answer
1k
views
A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation
very good implementation of A2C continuous for Pendulum-v0
Code has snippet to stop execution when mean of last 10 or 20 is higher than -20 but the results look like:
...
- 604
2
votes
1
answer
1k
views
How to scale or standardize data that is mostly 0 (ranges from 0-1)?
I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each ...
- 123
1
vote
0
answers
25
views
F-test for comparing the mean of two groups
What would be the procedure, in terms of F-test, if I would like to check if the mean of one group is greater than the other group (alternative hypothesis: $\mu_1 > \mu_2$)?
And also what would ...
3
votes
1
answer
2k
views
Plotting Weibull distribution on Wind Speed
I have a 720 hourly set of wind speed and wind direction data and I want to fit the Weibull Distribution on it. After searching for some time, I wrote the following code in Python to get my ...
11
votes
2
answers
6k
views
how to check the distribution of the training set and testing set are similar
I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
- 287
0
votes
1
answer
2k
views
Distribution of error values in linear regression vs logistic regression
Why do error values in linear regression have to be normally distributed and why not in logistic regression?
2
votes
0
answers
906
views
How to create a prediction interval with the fact that the residuals follow a specific distribution (in python)
I am looking at a software development pipeline where I am predicting the lead time of different products flowing through the pipeline.
After applying a boxcox transformation on the lead time (...
- 133
2
votes
1
answer
2k
views
Normal distribution instead of Logistic distribution for classification
Logistic regression, based on the logistic function $\sigma(x) =
\frac{1}{1 + \exp(-x)}$, can be seen as a hypothesis testing problem. Where the reference distribution is the standard Logistic ...
- 169
2
votes
2
answers
173
views
How to visualize change in a distribution with a few outliers that account for a very large percent of the total?
I'm working on an edtech product where some of our traffic lands on webpages about textbooks.
Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "...
- 121
1
vote
1
answer
133
views
Clustering by Distributions of Groups of observations
I have data as shown below. Like the groups shown A,B,C.. there are roughly 5400 groups of data and 15 observations for every group (I sampled 6-7 for each group and pasted them here).
...
- 11
1
vote
0
answers
19
views
How to elegantly caclulate probability distribution parameters for a particular random variable given some observed data?
I am using bnlearn package and R to learn Bayesian Network structure and also fit it using Maximum Likelihood estimation(MLE). bn.fit uses MLE to learn[as i understand] a generalized probability ...
- 11
6
votes
1
answer
6k
views
How can I plot/display a dataset or an image distribution?
I want to view a specific image or a dataset's distribution, and see if they are different.
Does simply writing something like :
...
- 565
1
vote
0
answers
364
views
Two-sample K–S test Kolmogorov–Smirnov test in Python
For my evaluation, I have three different time-series data of the following format with different characterstics where the first column is timestamp and the second column is the value.
...
- 207
1
vote
0
answers
131
views
Probability Distribution of the Process Duration Time based on the Process Status Measurements
I have data of execution of several processes. Each execution is identified by the process_id.
The duration of the execution is measured by external agents, testing ...
- 1,113
4
votes
3
answers
1k
views
How to predict whether or not a customer will renew
I have a dataset of customer contracts that specify a start date and if applicable an end date. Each month a customer is up for renewal. Below is an example of how the data is organized in excel:
<...
- 41
0
votes
1
answer
771
views
KL divergence in VAE
If I understand correctly KL-divergence is relative entropy of two distributions. To calculate KL divergence of two distributions, you would need two vectors of random variables.
What I do not ...
- 175
1
vote
0
answers
193
views
How to check the similarity between two transition matrixes
I have two transition matrixes in which the probabilities for the transition between each from state to each to states are ...
- 111
4
votes
2
answers
348
views
Standard Deviation for Z-scores
I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of:
$\sigma = \sqrt{p * n * (1-p)}$
My data is ...
- 2,099
1
vote
1
answer
238
views
Generating a set of different scenarios based on some initial observations
I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for ...
- 11
0
votes
2
answers
176
views
Maximum likelihood estimation vs calculating distribution parameters "manually"
I'm sorry for asking probably elementary question, but I cannot understand how estimating probability distribution parameters using maximum likelihood estimation method differs from calculating these ...
- 113
1
vote
1
answer
9k
views
What Does the Normalization Factor Mean in the AdaBoost Algorithm?
I am studying the AdaBoost algorithm. The update rule for a weak hypothesis is:
$Dt+1(i) = Dt(i)exp(−αtyiht(xi))/zt $
where $zt$ is a normalization factor chosen so that $Dt+1$ is a distribution.
...
- 135
1
vote
1
answer
35
views
Calculate whether datapoints are part of a larger distribution
I have some normally distributed variables (~800) and some variables that are in some way special (~30). I need to find out whether the special ones can be considered normal members of the ...
- 113
1
vote
1
answer
516
views
How often do we see normally distributed data
I am having difficulty in exactly understanding several statistical tests, such as the t-test and ANOVA test. These tests require that the data we use be normally distributed.
However, whilst ...
- 131
1
vote
1
answer
29
views
How to get maintenance interval from maintenance outcomes?
I have a machine, which needs maintenance.
Every time the technician visits the machine, there are four possible outcomes:
a) The machine is broken, b) The machine is still running, but it's high ...
- 11
1
vote
3
answers
191
views
Binomial family in logistic regression
I was asked in an interview why do we use the binomial distribution in logistic regression and how is it related to the class that we are predicting?
Could anyone explain, without any mathematical ...
- 111
1
vote
0
answers
80
views
On Noise Contrastive Estimation, replace noise distribution with difficult examples
In noise contrastive estimation, we learn a binary classifier to classify noise from the true distribution.
The problem I'm trying to solve has discrete input variables yet it's intractable to sum ...
- 904
2
votes
0
answers
209
views
Wasserstein distance between Gaussian and the empirical distribution
Wasserstein distance between two gaussians has a closed form solution. Does the same hold for the distance between a Gaussian with a fixed variance(say 1) and the empirical data distribution?
...
- 41
0
votes
1
answer
162
views
2
votes
1
answer
174
views
Does classification of a balanced data-set lead to any problem?
So I came across a bioinformatics paper, where I found a line which says:
One potential problem with using a training set with
equal numbers of positive and negative examples in cross-validation
...
- 1,161
2
votes
2
answers
453
views
can I say that my variable is "approximately" normally distributed?
I am studying a variable on 50000 observations, I have applied a cox box transformation to make it normal, but even with transformation the kolmogorov smirnof and the anderson darling test tells that ...
- 61
0
votes
1
answer
22
views
Feature has a pattern in relation to class but does not enhance classifier to predict class
I have a feature x that when I plot it again my class variable y it shows some sort of pattern, i.e. it is obvious that x has a relation with y, but when I add the x to my logistic model it reduce the ...
- 234
1
vote
2
answers
61
views
Algorithms that would benefit from variable transformations?
1- Which algorithms would benefit from data that has been transformed, so that distributions of continuous variables resemble that of a normal distribution ?
2- What would be the benefits of ...
- 403