Questions tagged [distribution]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

How to compare distribution of 2 continuous variable datasets

i want to compare 2 datasets and check for their similarity. I have tried statistical tests like ks test , z test but they gave a p value of 0.0 for most columns. I then read ks test won't work ...
user avatar
  • 1
0 votes
0 answers
7 views

the mean and standard deviation aren't the same as those of the input data i provided after sampling

I have a log-normal mean and a standard deviation. after i converted them to the underlying normal distribution's parameters mu and sigma, I sampled from the log-normal distribution however when i ...
user avatar
0 votes
0 answers
9 views

How to properly setup jensen_shannon_divergence and infinity norm in tensorflow data validation for skew and drift checks

Tensorflow data validation offers the capability of checking data skew and drift and the documentation also mention that "Setting the correct distance is typically an iterative process requiring ...
user avatar
  • 193
0 votes
0 answers
15 views

How to explain this coorelation between features

Can somebody please help how to explain this correlation between features, as it does not have linear coorelation, but still seems to have somewhat coorelation. Here is the screenshoot:
user avatar
2 votes
1 answer
17 views

How to measure statistical similarity or discrepancy between a dataset and a distribution?

Is any way to measure statistical similarity or discrepancy between a dataset and a distribution? I have do some research, but find most of method are intended to describe discrepancy between data and ...
user avatar
  • 21
0 votes
1 answer
20 views

Is it possible to find an internal event in a Classification between two classes?

I am very new to the machine learning area. So my question might be trivial  I have two classes $U, V$ of binary vectors. In the training phase,  I use $u_1,\ldots, u_{1000}$ from $U$ class and $v_1, \...
user avatar
  • 1
0 votes
0 answers
13 views

KDE Sampling with negative density and/or class-specific weighting

I have a dataset which contains two overlapping distributions/classes of points. I have been trying to sample from just one of these distributions/classes using the scikit learn Kernel Density class, ...
user avatar
  • 13
0 votes
0 answers
24 views

Function for KDE-style distribution generation for sampling

I have some points in pytorch and I would like to sample from a distribution that resembles these points. I noticed that the seaborn kde plots seem to draw out/define a distribution graphically and I ...
user avatar
  • 13
0 votes
0 answers
10 views

what level of discrepancy do I target for a good interpolation?

I'm performing some interpolation comparison and, of course, "the quality" of the training sample is a key parameter to survey. In this case I can create the dataset. For this reason, I try ...
user avatar
1 vote
1 answer
18 views

What thought processes do people use for generating priors on a variable's probability distribution?

Example Consider a block of text with a variety of sentence types within it, of which there are 7. Within a text these sentences will be more or less likely to appear, dependent on where in the text ...
user avatar
  • 133
0 votes
0 answers
30 views

Generating new sample with same distribution

I have a timeseries data for 1 week. The data contains readings from a device for certain hours of the day. There are about 8-10 readings per day at different timestamps. The timestamps recorded for ...
user avatar
  • 1
0 votes
0 answers
19 views

How to manage sampling bias between training data and real-world data?

I'm currently working on a binary classification problem. My training dataset is rather small with only 1000 elements. (I don't know if it is relevant : my problem is similar to the "spam ...
user avatar
  • 1
1 vote
1 answer
24 views

Handling gaps in regression model

I'm facing a regression problem where I'm supposed to predict the delay of some trains. There's some peculiar particularity, however: a train is not considered delayed until it has more than 10 mins ...
user avatar
0 votes
2 answers
138 views

Comparing distributions Python

I have a larger dataset (random variable) 'x' containing values approximating a Gaussian distribution. From 'x', a much smaller random variable 'y' is sampled without replacement. I want to compare ...
user avatar
  • 121
0 votes
0 answers
10 views

What kind of impact has a standardization on our ML model?

I understand that standardisation helps to compare two different normal distributions (e.g. performance of students in cambridge vs. stanford) and it also helps find probabilites by using the z score ...
user avatar
0 votes
0 answers
15 views

When do I need Statistical Signifcance testing and when not?

Hi there I have a handful of questions regarding statistical significance testing. As a newcomer I have sometimes topics that I do not really understand entirely. One of them is checking for ...
user avatar
1 vote
0 answers
13 views

Probability notation q(y) and q(Y) and its implication to vector functions

The function in question is (from Appendix B, Proof of proposition 2.1 from Posterior Regularization for Structured Latent Variable Models): $$q(\textbf{Z}) = \frac{p_{\theta}(\textbf{Z}|\textbf{X})...
user avatar
  • 131
0 votes
0 answers
10 views

Random number generator and JL lemma

Suppose we are given access to a random number generator U, which generates independent random real variables distributed uniformly in the range [0,1). Show that this can be used to produce an ...
user avatar
-1 votes
1 answer
23 views

Normalizing Distributions for features in predicting

Should features used for predictions be normalized if they are highly skewed. Or should I only normalize target feature that is supposed to be predicted?
user avatar
  • 33
0 votes
0 answers
18 views

How to do ? serious skewed right transformations normally distribution

My target is doing two-way Anova. But Here is some problem. This is my data like I try log(X). Looks like a normal distribution. But try Kolmogorov-Smirnov prove is not. Can anyone recommend another ...
user avatar
3 votes
2 answers
95 views

Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
user avatar
  • 359
1 vote
0 answers
14 views

Is there a way to create a continuous distribution graph from discrete percentiles?

I have an underlying distribution that I know is 1-tailed (Packet Response time), but the service that runs the benchmark is opaque to me. It only returns a percentile table up to some number of nines ...
user avatar
  • 11
0 votes
1 answer
17 views

Distribution of text data

How can I identify whether the training data and test data came from same distribution or not? I tried with TFIDF and cosine similarity ...
user avatar
0 votes
0 answers
11 views

Need a random process/distribution where I can pass a certain level of bias for producing an outcome

My first question here if am not clear please let me know. My objective a startup Sportsbook wants to test its algo to see how it manages game lines for incoming bets placed on a particular game. For ...
user avatar
2 votes
0 answers
10 views

Measuring the distance between data points based on mutual linkages

How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages? I don't know the technical term for that, so here is a fictitious example from scientific ...
user avatar
  • 123
1 vote
1 answer
39 views

How to find the distribution of a single variable based on population distribution

I am trying to find the age distribution of the subset of married people of a population. However, I realise that just visualising the married count by age (Figure 1) is highly dependent on the ...
user avatar
  • 11
1 vote
2 answers
59 views

Distribution Shift vs Transfer Learning

Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem [1] ...
user avatar
0 votes
0 answers
18 views

How to distribute train and test set equally in the matter of low and fast growth cases?

I'm working on a project which we are using python (Random Forest Regression) predicting tumor doubling time. when I plot different test sets I found that my data is not distributed equally into train ...
user avatar
0 votes
0 answers
18 views

How to perform a Monte Carlo simulation with continuous sampling using discrete quantiles?

Assume I have registered the duration of 10 tasks and built the table below with using this data: Duration For how many tasks it happened 4 days 5 task 6 days 2 task 8 days 2 task 10 days 1 task ...
user avatar
0 votes
0 answers
9 views

Use A Distribution To Measure Customer Success

I am working at a marketplace startup and one of our success metrics is average number of transactions. To better represent customer success I suggested that we leverage a distribution to measure the ...
user avatar
0 votes
0 answers
57 views

How to generate a positive skewed array?

As the title suggests, I would like to specify the task that I really need help with: Given three possible distributions of an array: I know that given a skewed dataset (the left and the right), then ...
user avatar
  • 1
0 votes
0 answers
32 views

Data Transformation for Machine Learning Regression Task

I am performing a ML regression task, using XGBoost Regressor. I am using financial time series data, namely the Close price of the EUR/USD exchange rate which I will transform into geometric log ...
user avatar
0 votes
0 answers
61 views

A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
user avatar
  • 791
1 vote
1 answer
39 views

Check if distribution per week is the same

I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week. Initial dataset Customer Date Sales Alpha 2019-02-23 527 ...
user avatar
  • 11
1 vote
0 answers
16 views

Criteria for assessing difficulty of a question

I have a list of questions and how many times they have been answered correctly and incorrectly. Based on this, I applied the formula: ...
user avatar
0 votes
0 answers
26 views

Predicting a distribution (instead of a point estimate) with supervised machine learning

I'm new to machine learning, but it seems that supervised learning algorithms that aren't also considered statistical models (random forests, neural networks, etc...) are focused on predicting a point ...
user avatar
1 vote
0 answers
29 views

Testing a Binary Classifier

I have been training a binary multilayer perceptron on a database made out of roughly 3600 0 values, and 4 1 values. Afterwards, I'm testing the MLP on a test set made out of 7 0 values and 7 1 ...
user avatar
1 vote
0 answers
73 views

Estimating the (normal?) distribution of the grayscale values of a region in camera footage

The problem: Given a small ($25\times25$) region on a $1080$p image extracted from camera footage of a railway system, I am trying to detect whether there is a train present or not. I proceed to take ...
user avatar
  • 111
0 votes
1 answer
39 views

Non IID variables and SVM Classifier

I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a ...
user avatar
0 votes
1 answer
287 views

How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
user avatar
  • 71
1 vote
0 answers
185 views

Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the ...
user avatar
  • 71
1 vote
2 answers
294 views

Label distribution over training, validation and test [closed]

I am wondering over whether the number of classes distributed over my training, validation, and test label affects the model.
user avatar
0 votes
1 answer
182 views

derivation for expected value for variance

Hi Im taking a course about probability distribution in datascience and below is derivation of the expected value for the variance Variance = expected value of the squared difference from mean for ...
user avatar
  • 1,131
0 votes
1 answer
24 views

The upper range of a collected dataset is most likely accurate, but the rest may suffer biased omissions: How to call this phenomenon?

Background: In collecting a dataset of a specific unit ordered by a numeric variable, it is possible that the upper 'cloud' of the dataset is correct, while the 'tail' seems inaccurate. I can thus ...
user avatar
  • 123
0 votes
1 answer
25 views

Individual models gives quite same distribution on Test set, whereas Ensembling gives better result but very different distribution

I am working on a binary classification problem with unbalanced data (17% for positive class). The problem is as following: My three individual models when predicting on the test set (for which I don'...
user avatar
1 vote
0 answers
30 views

Normalizing variables with logarithmic shape

A simple model with two variables [A,B] to train, let's say, a logistic regression or any other classification model: A: Flat distribution from 0 to 100. B: A logarithmic distribution from 0 to a few ...
user avatar
1 vote
1 answer
35 views

How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

I am researching on pay-scales, and wish to receive advise to treat data of salaries. Objective My interest is to approximate the salary corresponding to different hierarchical levels in an ...
user avatar
3 votes
0 answers
37 views

Non-Gaussian like distributions - Classifier of source data fails on target data

I ask you for help on a classification problem (classes are represented by the numbers 0,1 and 2). All features are extracted from time series data (fundamental is sinus shape). I have a source ...
user avatar
  • 41
1 vote
1 answer
378 views

Normal vs Uniform Distribution for machine learning

I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is ...
user avatar
1 vote
1 answer
95 views

distribution difference between image and text

Once for the task of image captioning I've read that, the features extracted from image and text by deep networks are from two different worlds and got different distribution. My question is how is ...
user avatar