Skip to main content

Questions tagged [distribution]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
1 answer
112 views

How to combine data having similar distribution?

I have a collection of time series data with data points of around 2 years of daily data. I am thinking of a way to increase the number of data points in it so that the neural network gets a better ...
0 votes
0 answers
18 views

Probability distribution for lstm and mlp

I have total of 6300 samples, 5800 of which are training data, and 500 of which are testing data. We compare the performance of LSTM and multilayer perceptron (MLP) with one hidden layer in terms of ...
1 vote
1 answer
220 views

How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
4 votes
2 answers
688 views

Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
2 votes
1 answer
55 views

How to measure statistical similarity or discrepancy between a dataset and a distribution?

Is any way to measure statistical similarity or discrepancy between a dataset and a distribution? I have do some research, but find most of method are intended to describe discrepancy between data and ...
1 vote
3 answers
184 views

Binomial family in logistic regression

I was asked in an interview why do we use the binomial distribution in logistic regression and how is it related to the class that we are predicting? Could anyone explain, without any mathematical ...
0 votes
1 answer
94 views

Non IID variables and SVM Classifier

I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
6 votes
3 answers
4k views

Boxplots or violinplots?

This is quite a general question, perhaps somewhat opinion-based. In most papers people use boxplots to visualize a certain distribution, yet violinplots are able to give more information. ...
0 votes
1 answer
226 views

Getting a balanced sample across many variables

Let’s say each element in my population has several attributes. Let’s call then A, B, C, D, E, F. Let’s say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30). ...
0 votes
1 answer
1k views

derivation for expected value for variance

Hi Im taking a course about probability distribution in datascience and below is derivation of the expected value for the variance Variance = expected value of the squared difference from mean for ...
1 vote
1 answer
209 views

Generating a set of different scenarios based on some initial observations

I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for ...
2 votes
1 answer
1k views

Normal vs Uniform Distribution for machine learning

I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is ...
1 vote
1 answer
62 views

Finding the worst affected industry due to COVID in terms of unemployment

My goal is to find the worst affected industries from COVID—19 in terms unemployment. In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each ...
1 vote
2 answers
60 views

Is there a measure to compare features on the basis of normality

I have a dataset of cars and it has many features including 'acceleration’, ‘horsepower’, and ‘mpg'. I am supposed to check which of these features is the most similar to a normal distribution, so I ...
0 votes
1 answer
74 views

difference btw data distribution and frquency distribution

I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?
2 votes
1 answer
129 views

Co-joining multi-peak histograms

I am analysing a bunch of data files which represent responsiveness of cells to addition of a drug. If a drug is not added, cell responds normally, if it is added, it shows abnormal patterns: , . We ...
1 vote
1 answer
89 views

How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0 votes
1 answer
71 views

Using Maximum Mean Discrepancy (MMD) to Compare Kernel Density Estimates (KDEs)

I'm interested in comparing two Kernel Density Estimates (KDEs) and I've come across the Maximum Mean Discrepancy (MMD) metric ...
2 votes
3 answers
1k views

A/B Testing (Binomial Distribution vs Random Distribution)

When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
0 votes
1 answer
54 views

Is Maximum Mean Discrepancy (MMD) suitable for comparing distributions with different sample sizes?

I'm working on a project where I need to compare the similarity of two probability distributions using MMD. However, the two datasets have different sample sizes. I've read that MMD can be affected by ...
0 votes
0 answers
14 views

Two overlap gaussian mixtures come from one or two generations of data?

Reviewing for a ML exam, my teacher asked last year a philosophical question about gaussians and generations of data which I couldn’t find no good answer. Imagine that an oracle can say with 100% ...
0 votes
1 answer
35 views

Binary Classification Dataset Distribution

I have a question regarding the distribution of my dataset. I have a binary classification task which I need to label. I now want to know how I should distribute the labels for optimal training / ...
0 votes
0 answers
26 views

What can I do when validation data and test data has different distribution in imbalance classification?

I am building classification model for bio (scRNA) data. Datasets in this field, for example, dataset A has 1, 2 classes, dataset B has 2, 3 classes kind of that. So I integrated datasets for training ...
0 votes
0 answers
31 views

Bayesian Linear regression for non-gaussian distribution

Which distribution should be considered for non-gaussian distribution while building Bayesian Linear regression. I am trying to build this model using Pymc3 library but since I have just started to ...
0 votes
0 answers
8 views

Grouped Time Restricted Demand Regression with value cap

so I am working on quite an interesting regression task that I haven't encountered before. Our company sells products (steel) in tons. We offer contracts where the customer orders a certain amount of ...
2 votes
2 answers
1k views

Does t-test require Standard Deviation of sample for calculation

Might be a novice question, but the main difference between a t-test and z-test, I was able to understand, is that the z-test calculation requires the SD value of the population where as in a t-test, ...
0 votes
1 answer
363 views

What is a good metric for comparing a single value to a distribution?

I'd like to compare single values from one distribution to another distribution, effectively transforming the second distribution in such way, that its values reflect both datasets simultaneously, i.e....
0 votes
0 answers
21 views

What distribution-shift datasets exist for ML?

What datasets there are which contain a (labelled) distribution shift or which distribution-shifted datasets share classes? Currently I have found some interesting results using CIFAR10 and the ...
0 votes
0 answers
50 views

Train neural network to predict multiple distributions

I aim to train a neural network to predict 2 distributions (10 quantiles, i.e. deciles) at 5 time points. So my y is of shape: ...
0 votes
0 answers
19 views

Using neural networks to learn total distributions from linearly transformed samples

I am trying to explore the use of neural network based models to replace a cumbersome challenge with a faster more approximate method. The problem involves learning a distribution termed the "...
4 votes
1 answer
2k views

What is the difference between Covariate Shift, Label Shift, Concept Shift, Concept Drift, and Prior Probability Shift?

As a beginner in MLOps, I was overwhelmed by some confusing definitions. As far as I understand, when we have a classifier or regressor with y = f(X) function: <...
0 votes
2 answers
71 views

How do I fix mis-rendered matplotlib?

How do I correct my data or format it so that it is presentable, and fix my graphs? Dataset is 345551 rows × 7 columns. I am using numpy, pandas, seaborn and matplot lib. It seems that my pricing ...
1 vote
2 answers
2k views

Data Analytics how to read ECDF graph

Hi there, My question is about how to read ECDF graphs. I am still quite unsure what the jumps / zig-zags in the graph mean and what is happening when there is a horizontal line and so on. I would be ...
0 votes
2 answers
2k views

The tag on feature-scaling says "zero mean" and unit variance. Is it correct?

The tag on feature scaling says: Popular feature scaling types include scaling the data to have zero mean and unit variance, and scaling the data between a given minimum and maximum value. It is ...
0 votes
1 answer
62 views

For standard deviation's formula, why does division by sample size come before square rooting?

Why is this used to calculate a sample's standard deviation: $s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N(x_i-\bar{x})^2}$ and not something like: $s=\frac{1}{N-1}\sqrt{\sum_{i=1}^N(x_i-\bar{x})^2}$ I ...
1 vote
2 answers
123 views

Univariate Outlier Detection

Let's say I have a dataset with the following format: customerid product orders_in_last7days orders_in_last6days orders_in_last5days orders_in_last4days orders_in_last3days orders_in_last2days ...
0 votes
2 answers
2k views

How to plot categorical variables with a pie chart

I am concerned with a single column (fruit) from my df: ...
0 votes
0 answers
64 views

Distribution check - different results using plt.hist and seaborn.displot

I have an apparently very simple task: checking the distribution of values contained in an array data. Only thing is, I get two completely different results by ...
1 vote
1 answer
59 views

What is the distribution of the goal feature in this dataset

https://www.kaggle.com/kemical/kickstarter-projects I was checking the distribution of the "goal" feature in this 2016 KickStarter data set. I figure goal would be more normally distributed ...
6 votes
4 answers
3k views

Regression: How to deal with positive skewness in continuous target variable

I'm working on a regression problem. My aim is to "learn" the distribution of a continuous target $y$ as good as possible to make predictions. My model looks like: $$y_i=\beta X_i + u_i.$$ $y$ is ...
1 vote
0 answers
60 views

Understanding Syntactic divergence

I am trying to read a paper https://arxiv.org/abs/2004.14444 Section 6.1 of the paper describes Syntactic divergence. I have confusion regarding the distribution graphs and split of the dataset. If a ...
1 vote
0 answers
25 views

Strange behaviour around zero for predicted distributions from a deep learning regression model

Can somebody help make sense of these very odd distributions that I obtained from my trained deep learning regression model? The model was trained with either MAE or MSE loss, which is what the ...
0 votes
1 answer
98 views

What is the next after finding best distribution for my data?

I found the distribution of my data with "distfit" library for python. But what now? The best distribution that describes my data is "weibull" distribution. But I don't know what ...
0 votes
1 answer
889 views

How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
0 votes
0 answers
51 views

Multiclass classification (gradient boosted trees) predictions distribution using softmax()

Let's consider a multiclass problem where the target is composed by 20% class 'A', 50% of class 'B' and 30% of class 'C'. The model is trained and then the class predictions are obtained via the ...
0 votes
1 answer
105 views

How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?

I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following: from this plot I see that ...
0 votes
1 answer
199 views

Compute the pdf from pandas kde

I have data (features/targets in machine learning terminology), e.g. X1(t), X2(t), ... XN(t) and dependent variable y(t). I can use pandas to plot the kde's of the independent variables (X1(t),...). I ...
7 votes
2 answers
14k views

Plotting different values in pandas histogram with different colors

I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used <...
1 vote
1 answer
48 views

name of log(n+1) plot

I am trying to plot a distribution of positive integers which contains a lot of variance. I opted to use the log of the y-values but that causes issues due to the inclusion of zeros. I though of ...
0 votes
0 answers
28 views

Task Distribution

I would like to distribute tasks (going to companies and signing a deal) to sales representatives based on some condition like company location (is it near the representative's place) or profit of the ...