Skip to main content

Questions tagged [distribution]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
10 votes
2 answers
6k views

how to check the distribution of the training set and testing set are similar

I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
Bowen Peng's user avatar
7 votes
3 answers
5k views

xgboost: Is there a way to perform regression on rates/percentages data?

I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific ...
Coolio2654's user avatar
7 votes
2 answers
14k views

Plotting different values in pandas histogram with different colors

I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used <...
enterML's user avatar
  • 3,041
7 votes
3 answers
1k views

Which outlier detection can detect these outliers?

I have a vector and want to detect outliers in it. The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal. ...
Arkan's user avatar
  • 453
6 votes
3 answers
4k views

Boxplots or violinplots?

This is quite a general question, perhaps somewhat opinion-based. In most papers people use boxplots to visualize a certain distribution, yet violinplots are able to give more information. ...
Archie's user avatar
  • 873
6 votes
1 answer
6k views

How can I plot/display a dataset or an image distribution?

I want to view a specific image or a dataset's distribution, and see if they are different. Does simply writing something like : ...
Hossein's user avatar
  • 565
6 votes
4 answers
3k views

Regression: How to deal with positive skewness in continuous target variable

I'm working on a regression problem. My aim is to "learn" the distribution of a continuous target $y$ as good as possible to make predictions. My model looks like: $$y_i=\beta X_i + u_i.$$ $y$ is ...
Peter's user avatar
  • 7,526
6 votes
1 answer
2k views

How to estimate the mutual information numerically?

Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ? ...
patapouf_ai's user avatar
5 votes
2 answers
6k views

Working with Data which is not Normal/Gaussian

What happens if my data/feature is not normal? Can I still use machine learning algorithms to utilize such data for predictions? I noticed in many data sciences courses, there is always a strong ...
Newbie01's user avatar
4 votes
1 answer
2k views

What is the difference between Covariate Shift, Label Shift, Concept Shift, Concept Drift, and Prior Probability Shift?

As a beginner in MLOps, I was overwhelmed by some confusing definitions. As far as I understand, when we have a classifier or regressor with y = f(X) function: <...
Mohsen Mahmoodzadeh's user avatar
4 votes
1 answer
3k views

Why do train, test, validation datasets need to have the same distribution?

I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution. Can ...
zaza's user avatar
  • 141
4 votes
2 answers
3k views

Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?

I'm looking at telecom customers data. Two of the variables I'm looking at currently are: Monthly Charges - The total amount charged to the customer monthly. Is Senior Citizen - Whether the customer ...
helloworld's user avatar
4 votes
3 answers
1k views

How to predict whether or not a customer will renew

I have a dataset of customer contracts that specify a start date and if applicable an end date. Each month a customer is up for renewal. Below is an example of how the data is organized in excel: <...
Geometric's user avatar
4 votes
2 answers
59 views

Is it possible to train probabilistic model to return several distributions?

I have nonlinear data of function y(x), which is let's say parabolic. At some points of x there are several y's (look at the picture). Is it possible to train a probabilistic model to return several ...
BatyaGG's user avatar
  • 141
4 votes
2 answers
335 views

Standard Deviation for Z-scores

I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of: $\sigma = \sqrt{p * n * (1-p)}$ My data is ...
I_Play_With_Data's user avatar
4 votes
3 answers
14k views

Transform a skewed distribution into a Gaussian distribution

I have a skewed distribution that looks like this: How can I transform it to a Gaussian distribution? The values represent ranks, so modifying the values does not cause information loss as long as ...
bkoodaa's user avatar
  • 323
4 votes
2 answers
686 views

Analysis of probability distribution of each features and Machine Learning

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis. However, it is not obvious to me ...
Student's user avatar
  • 411
4 votes
0 answers
84 views

Non-Gaussian like distributions - Classifier of source data fails on target data

I ask you for help on a classification problem (classes are represented by the numbers 0,1 and 2). All features are extracted from time series data (fundamental is sinus shape). I have a source ...
deniz's user avatar
  • 51
3 votes
1 answer
942 views

Confidence value for face recognition

In the context of face recognition I have the following histogram: blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the ...
lorenzo's user avatar
  • 131
3 votes
1 answer
2k views

Normal distribution and Random Forest

I have big table in dataframe (600k rows) which has y column (the variable I want to predict) and other 4 other columns that are the X. I have run RF regressor and I got score of 0.87 when I run it ...
Reut's user avatar
  • 349
3 votes
1 answer
2k views

Plotting Weibull distribution on Wind Speed

I have a 720 hourly set of wind speed and wind direction data and I want to fit the Weibull Distribution on it. After searching for some time, I wrote the following code in Python to get my ...
cwanderroycbooks's user avatar
3 votes
1 answer
336 views

Shifted feature distribution across different datasets

I am trying to validate a classifier using two different training and testing datasets. The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. ...
gc5's user avatar
  • 879
3 votes
2 answers
48 views

evaluation metrics for multiple values per session

I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based ...
sbr's user avatar
  • 131
2 votes
3 answers
201 views

How this visualisation was made?

I really like how this visualization represents the survey participants. Is any tool for that? (Or R/python library?)
Bálint Kőszegi's user avatar
2 votes
2 answers
445 views

can I say that my variable is "approximately" normally distributed?

I am studying a variable on 50000 observations, I have applied a cox box transformation to make it normal, but even with transformation the kolmogorov smirnof and the anderson darling test tells that ...
youngam's user avatar
  • 61
2 votes
3 answers
2k views

Is there a cost associated with converting Koalas dataframe to Spark dataframe?

I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
DataBach's user avatar
  • 165
2 votes
2 answers
956 views

Fitting a distribution to data

I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation ...
TaxpayersMoney's user avatar
2 votes
1 answer
2k views

Normal distribution instead of Logistic distribution for classification

Logistic regression, based on the logistic function $\sigma(x) = \frac{1}{1 + \exp(-x)}$, can be seen as a hypothesis testing problem. Where the reference distribution is the standard Logistic ...
HOANG GIANG's user avatar
2 votes
1 answer
1k views

How to resample one dataset to conform to the distribution of another dataset?

I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features?? ...
cybergeek654's user avatar
2 votes
1 answer
83 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
Muhammad Ali's user avatar
  • 2,487
2 votes
2 answers
170 views

How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

I'm working on an edtech product where some of our traffic lands on webpages about textbooks. Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "...
samthebrand's user avatar
2 votes
1 answer
172 views

Does classification of a balanced data-set lead to any problem?

So I came across a bioinformatics paper, where I found a line which says: One potential problem with using a training set with equal numbers of positive and negative examples in cross-validation ...
girl101's user avatar
  • 1,161
2 votes
1 answer
55 views

How to measure statistical similarity or discrepancy between a dataset and a distribution?

Is any way to measure statistical similarity or discrepancy between a dataset and a distribution? I have do some research, but find most of method are intended to describe discrepancy between data and ...
nick's user avatar
  • 21
2 votes
2 answers
1k views

Does t-test require Standard Deviation of sample for calculation

Might be a novice question, but the main difference between a t-test and z-test, I was able to understand, is that the z-test calculation requires the SD value of the population where as in a t-test, ...
Chris_007's user avatar
  • 193
2 votes
3 answers
1k views

A/B Testing (Binomial Distribution vs Random Distribution)

When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
DD.'s user avatar
  • 121
2 votes
2 answers
184 views

How to generate a random sample and distribute values based in an probability distribution?

I want to generate a random sample based on this probability distribution: The line is the KDE of the histogram. My random sample will have n values, the value is ...
neves's user avatar
  • 151
2 votes
1 answer
291 views

Synthetic time series generation according to some distribution

I'm trying to develop a change detection model that uses sliding windows. Given a time series with some features I've a sliding widows that analyses that time period and compares with a successive ...
exrezzo's user avatar
  • 123
2 votes
1 answer
36 views

Testing for gender composition between groups

I did kmeans with k=4 and I would like to find out if there are any differences in the composition of gender between the 4 clusters. Do I use Fisher's Exact Test of Independence? Is there any ...
TYL's user avatar
  • 171
2 votes
2 answers
16k views

When to use t-distribution instead of normal distribution?

According to the Student's t-distribution wiki article the t-distribution is used instead of the Normal distribution "when estimating the mean of a normally distributed population in situations where ...
thinwybk's user avatar
  • 193
2 votes
1 answer
218 views

Stratified Sampling Variable Choice

I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set. Is ...
Sandra Maria Nawar's user avatar
2 votes
2 answers
360 views

Machine learning learn to work well on future data distribution?

This is based on my limited machine learning scope and experience, so correct me if I'm wrong. Many of the currently used machine learning models (SVMs, boosted trees, DNNs) work under the assumption ...
Roy's user avatar
  • 289
2 votes
1 answer
808 views

Methods / Algorithms for rank scales based on cumulative scoring

Say you have an organization that requires employees to participate in a Q&A site similar to StackOverflow - questions and answers are voted upon, selected answers get extra points, certain ...
SB2055's user avatar
  • 73
2 votes
0 answers
18 views

Measuring the distance between data points based on mutual linkages

How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages? I don't know the technical term for that, so here is a fictitious example from scientific ...
anpami's user avatar
  • 123
2 votes
1 answer
1k views

Normal vs Uniform Distribution for machine learning

I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is ...
Michael Pulis's user avatar
2 votes
0 answers
111 views

How to estimate the marginal distribution of a class with respect to one predictor in a classification task?

I have a dataset with a binary dependent variable $y \in \{0,1\}$ and a set of predictors $x1,x2,..,t$. Here, $t$ is the time in minutes (in 24 hrs, that is $t \in (0,1440)$). I want to estimate the ...
Shanthan K's user avatar
2 votes
0 answers
224 views

Use of PYMC distribution.dist()

In pymc3 documentation, it specifies that the .dist() member for distribution allows the distribution to be used without a model for sampling and use of the logp functions e.g. ...
rocklegend's user avatar
2 votes
1 answer
129 views

Co-joining multi-peak histograms

I am analysing a bunch of data files which represent responsiveness of cells to addition of a drug. If a drug is not added, cell responds normally, if it is added, it shows abnormal patterns: , . We ...
Jericho Jones's user avatar
2 votes
0 answers
812 views

difference between normal skewed distribution and skewed distribution [closed]

From what I've read normal skewed is a distribution that has all the properties of normal distribution and is skewed: according to this resource:https://www.statisticshowto.datasciencecentral.com/...
haneulkim's user avatar
  • 469
2 votes
1 answer
111 views

How to combine data having similar distribution?

I have a collection of time series data with data points of around 2 years of daily data. I am thinking of a way to increase the number of data points in it so that the neural network gets a better ...
vignesh_md's user avatar
2 votes
1 answer
797 views

How to scale or standardize data that is mostly 0 (ranges from 0-1)?

I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each ...
StonedTensor's user avatar