Questions tagged [distribution]
The distribution tag has no usage guidance.
165
questions
0
votes
1
answer
10
views
Compute the pdf from pandas kde
I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...).
I ...
- 71
0
votes
1
answer
32
views
How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?
I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following:
from this plot I see that ...
0
votes
0
answers
13
views
Task Distribution
I would like to distribute tasks (going to companies and signing a deal) to sales representatives based on some condition like company location (is it near the representative's place) or profit of the ...
- 1
1
vote
1
answer
34
views
name of log(n+1) plot
I am trying to plot a distribution of positive integers which contains a lot of variance. I opted to use the log of the y-values but that causes issues due to the inclusion of zeros. I though of ...
- 13
0
votes
1
answer
38
views
How to plot categorical variables with a pie chart
I am concerned with a single column (fruit) from my df:
...
0
votes
0
answers
8
views
Split 2 zipfian distributions
Suppose I have 2 or more sequences distributed by Zipf's law merged together. It would look like this on log-log graph after sorting:
or like this:
What would be the best way to split them? I cannot ...
- 101
1
vote
0
answers
14
views
How can PCA be distributed among workers?
I want to distribute the work from PCA among a set of workers.
So let samples $x_1,... , x_d\in \mathbb{R}^n$ be samples. Then to find a dimension reduction subspace we need
covariance matrix $cov(\...
- 111
0
votes
0
answers
8
views
Could someone explain how to interpret distributions in tensorboard?
Could someone explain how to interpret these graphs which are obtained after running a NN model in distributions tab in Tensorboard? Can these be used to identify whether my ML model is working well?
- 1
0
votes
0
answers
19
views
Generating data
I have been stuck in a Problem, for more than 2 weeks now.
I need to generate data set, that have actual Probability.
I am doing an experiment, my ML framework gives me probability intervals for the ...
- 1
0
votes
0
answers
39
views
How does factoring probability distributions help?
I'm trying to make sense of machine learning and am reading Deep Learning by Goodfellow, Bengio and Courville. In section 3.14 they show an example of factoring a distribution then say "These ...
- 1
0
votes
1
answer
10
views
Pedagogical examples relating to neural networks and statistics (distributions) of inputs and outputs?
The way I was taught and commonly learn about various neural network technologies is by looking at the problem from the "specific example" stand point. For example, if I were to research ...
- 123
0
votes
0
answers
6
views
Search one 2D distribution for point cluster most similar to another 2D distribution
Given a hand drawn constellation (2d distribution of points) and a map of all stars, how would you find the actual star distribution most similar to the drawn distribution?
If it's helpful, suppose we ...
- 184
0
votes
0
answers
13
views
How to compare distribution of 2 continuous variable datasets
i want to compare 2 datasets and check for their similarity. I have tried statistical tests like ks test , z test but they gave a p value of 0.0 for most columns. I then read ks test won't work ...
- 1
0
votes
0
answers
9
views
the mean and standard deviation aren't the same as those of the input data i provided after sampling
I have a log-normal mean and a standard deviation. after i converted them to the underlying normal distribution's parameters mu and sigma, I sampled from the log-normal distribution however when i ...
0
votes
0
answers
16
views
How to properly setup jensen_shannon_divergence and infinity norm in tensorflow data validation for skew and drift checks
Tensorflow data validation offers the capability of checking data skew and drift and the documentation also mention that "Setting the correct distance is typically an iterative process requiring ...
- 193
0
votes
0
answers
16
views
How to explain this coorelation between features
Can somebody please help how to explain this correlation between features, as it does not have linear coorelation, but still seems to have somewhat coorelation. Here is the screenshoot:
- 11
2
votes
1
answer
20
views
How to measure statistical similarity or discrepancy between a dataset and a distribution?
Is any way to measure statistical similarity or discrepancy between a dataset and a distribution? I have do some research, but find most of method are intended to describe discrepancy between data and ...
- 21
0
votes
1
answer
22
views
Is it possible to find an internal event in a Classification between two classes?
I am very new to the machine learning area. So my question might be trivial
I have two classes $U, V$ of binary vectors. In the training phase,
I use $u_1,\ldots, u_{1000}$ from $U$ class and $v_1, \...
- 1
0
votes
0
answers
17
views
KDE Sampling with negative density and/or class-specific weighting
I have a dataset which contains two overlapping distributions/classes of points. I have been trying to sample from just one of these distributions/classes using the scikit learn Kernel Density class, ...
- 45
0
votes
0
answers
36
views
Function for KDE-style distribution generation for sampling
I have some points in pytorch and I would like to sample from a distribution that resembles these points. I noticed that the seaborn kde plots seem to draw out/define a distribution graphically and I ...
- 45
0
votes
0
answers
10
views
what level of discrepancy do I target for a good interpolation?
I'm performing some interpolation comparison and, of course, "the quality" of the training sample is a key parameter to survey. In this case I can create the dataset. For this reason, I try ...
- 73
1
vote
1
answer
20
views
What thought processes do people use for generating priors on a variable's probability distribution?
Example
Consider a block of text with a variety of sentence types within it, of which there are 7. Within a text these sentences will be more or less likely to appear, dependent on where in the text ...
- 133
0
votes
0
answers
40
views
Generating new sample with same distribution
I have a timeseries data for 1 week. The data contains readings from a device for certain hours of the day. There are about 8-10 readings per day at different timestamps. The timestamps recorded for ...
- 1
0
votes
0
answers
21
views
How to manage sampling bias between training data and real-world data?
I'm currently working on a binary classification problem.
My training dataset is rather small with only 1000 elements.
(I don't know if it is relevant : my problem is similar to the "spam ...
- 1
1
vote
1
answer
27
views
Handling gaps in regression model
I'm facing a regression problem where I'm supposed to predict the delay of some trains. There's some peculiar particularity, however: a train is not considered delayed until it has more than 10 mins ...
- 65
0
votes
2
answers
666
views
Comparing distributions Python
I have a larger dataset (random variable) 'x' containing values approximating a Gaussian distribution. From 'x', a much smaller random variable 'y' is sampled without replacement. I want to compare ...
- 121
0
votes
0
answers
10
views
What kind of impact has a standardization on our ML model?
I understand that standardisation helps to compare two different normal distributions (e.g. performance of students in cambridge vs. stanford) and it also helps find probabilites by using the z score ...
- 1
0
votes
0
answers
15
views
When do I need Statistical Signifcance testing and when not?
Hi there I have a handful of questions regarding statistical significance testing.
As a newcomer I have sometimes topics that I do not really understand entirely.
One of them is checking for ...
- 1
1
vote
0
answers
16
views
Probability notation q(y) and q(Y) and its implication to vector functions
The function in question is (from Appendix B, Proof of proposition 2.1 from Posterior Regularization for Structured Latent Variable Models):
$$q(\textbf{Z}) = \frac{p_{\theta}(\textbf{Z}|\textbf{X})...
- 131
0
votes
0
answers
10
views
Random number generator and JL lemma
Suppose we are given access to a random number generator U, which generates
independent random real variables distributed uniformly in the range [0,1). Show
that this can be used to produce an ...
-1
votes
1
answer
26
views
Normalizing Distributions for features in predicting
Should features used for predictions be normalized if they are highly skewed. Or should I only normalize target feature that is supposed to be predicted?
- 33
0
votes
0
answers
21
views
How to do ? serious skewed right transformations normally distribution
My target is doing two-way Anova. But Here is some problem.
This is my data like
I try log(X). Looks like a normal distribution. But try Kolmogorov-Smirnov prove is not.
Can anyone recommend another ...
- 1
3
votes
2
answers
142
views
Analysis of probability distribution of each features and Machine Learning
While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis.
However, it is not obvious to me ...
- 359
1
vote
0
answers
14
views
Is there a way to create a continuous distribution graph from discrete percentiles?
I have an underlying distribution that I know is 1-tailed (Packet Response time), but the service that runs the benchmark is opaque to me. It only returns a percentile table up to some number of nines ...
- 11
0
votes
1
answer
21
views
Distribution of text data
How can I identify whether the training data and test data came from same distribution or not?
I tried with TFIDF and cosine similarity
...
- 209
0
votes
0
answers
11
views
Need a random process/distribution where I can pass a certain level of bias for producing an outcome
My first question here if am not clear please let me know.
My objective a startup Sportsbook wants to test its algo to see how it manages game lines for incoming bets placed on a particular game. For ...
- 101
2
votes
0
answers
13
views
Measuring the distance between data points based on mutual linkages
How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages?
I don't know the technical term for that, so here is a fictitious example from scientific ...
- 123
1
vote
1
answer
102
views
How to find the distribution of a single variable based on population distribution
I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
- 31
1
vote
2
answers
105
views
Distribution Shift vs Transfer Learning
Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem [1]
...
- 5,621
0
votes
0
answers
19
views
How to distribute train and test set equally in the matter of low and fast growth cases?
I'm working on a project which we are using python (Random Forest Regression) predicting tumor doubling time. when I plot different test sets I found that my data is not distributed equally into train ...
0
votes
0
answers
282
views
How to generate a positive skewed array?
As the title suggests, I would like to specify the task that I really need help with:
Given three possible distributions of an array:
I know that given a skewed dataset (the left and the right), then ...
- 1
0
votes
0
answers
90
views
A feature is still right-skewed after log scaling. How should it be normalized for machine learning?
I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
- 801
1
vote
1
answer
41
views
Check if distribution per week is the same
I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week.
Initial dataset
Customer
Date
Sales
Alpha
2019-02-23
527
...
- 11
1
vote
0
answers
17
views
Criteria for assessing difficulty of a question
I have a list of questions and how many times they have been answered correctly and incorrectly. Based on this, I applied the formula:
...
- 11
1
vote
0
answers
29
views
Testing a Binary Classifier
I have been training a binary multilayer perceptron on a database made out of roughly 3600 0 values, and 4 1 values. Afterwards, I'm testing the MLP on a test set made out of 7 0 values and 7 1 ...
1
vote
0
answers
73
views
Estimating the (normal?) distribution of the grayscale values of a region in camera footage
The problem:
Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
- 111
0
votes
1
answer
47
views
Non IID variables and SVM Classifier
I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
- 101
0
votes
1
answer
364
views
How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?
np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
- 71
1
vote
0
answers
270
views
Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?
I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
- 71
1
vote
2
answers
376
views
Label distribution over training, validation and test [closed]
I am wondering over whether the number of classes distributed over my training, validation, and test label affects the model.