9 votes
Accepted

Plotting different values in pandas histogram with different colors

There isn't any built-in function to do this directly in pandas, but by getting the array collection of AxesSubplot, iterating on them to retrieve the matplotlib ...
Julien Marrec's user avatar
8 votes
Accepted

Working with Data which is not Normal/Gaussian

There are models that do not make assumption that the underlying data distribution is a normal distribution. For example, support vector machine just cares about the boundaries of the separating ...
Siong Thye Goh's user avatar
7 votes
Accepted

how to check the distribution of the training set and testing set are similar

It looks like the person that wrote the blog is combining the samples from the test set and train set into one dataframe and then predicting if each sample comes from the test set or training set (his ...
fractalnature's user avatar
7 votes

xgboost: Is there a way to perform regression on rates/percentages data?

In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate ...
Peter's user avatar
  • 7,426
5 votes

Boxplots or violinplots?

I am a big fan of violin-plots. Although both aim for the same goal (visualizing distributions and key figures for that), box-plots have their limitations. Please have a look into following gif [1]): ...
Michael Dorner's user avatar
5 votes
Accepted

How can I plot/display a dataset or an image distribution?

I want to view a specific image or a dataset's distribution, and see if they are different. Does this do the trick? It depends what you want to understand or learn about your data. what does ...
n1k31t4's user avatar
  • 14.8k
5 votes

xgboost: Is there a way to perform regression on rates/percentages data?

You could use the reg:logistic objective function. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters Edit: You need to use either ...
Ben Reiniger's user avatar
  • 11.6k
4 votes

Which outlier detection can detect these outliers?

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor. ...
Marmite Bomber's user avatar
4 votes

How to estimate the mutual information numerically?

Binning One easy way to do such an estimate is to put the continuous values into bins and obtain a discrete problem. Split up the domains of $X$ and $Y$ into bins and count the number of points that ...
mrmcgreg's user avatar
  • 296
4 votes
Accepted

Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?

The first returns a probability density of the distributions. As you can see, they integrate to 1, i.e. they cover the same area (because they are probabilities, not the raw data). The second returns ...
Leevo's user avatar
  • 6,165
4 votes

How this visualisation was made?

My go to library would be matplotlib, with which it is relatively easy to generate something similar. I don't have the correct font family to render the exact output as above, but this hopefully ...
Anandologist's user avatar
4 votes

Why do train, test, validation datasets need to have the same distribution?

You can think of the data distribution as the world that the model lives in. You want to train it to perform well in this world and to do this you have to train it on examples representing this world. ...
matthiaw91's user avatar
  • 1,535
4 votes

Regression: How to deal with positive skewness in continuous target variable

Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions. Some takeways: Add count of values of features as ...
Noah Weber's user avatar
  • 5,669
4 votes

What is the difference between Covariate Shift, Label Shift, Concept Shift, Concept Drift, and Prior Probability Shift?

I am paraphrasing from the book "Designing Machine Learning Systems by Chip Huyen". In a supervised machine learning problem, the training dataset can be viewed as a set of samples from a ...
monte's user avatar
  • 141
3 votes

How often do we see normally distributed data

Your confusion is apt. Normally distributed data doesn't show up that often. Most of the real world datasets are more complex than normal. Many of natural occurring phenomenon (think: height of people ...
hssay's user avatar
  • 1,998
3 votes

Boxplots or violinplots?

Hugely dependent on both user and audience preference (violin plots being more unusual could throw people), so it is mostly up to you. One main reason to go for a violin plot is to give more detail ...
jshep's user avatar
  • 393
3 votes

How to predict whether or not a customer will renew

What you are trying to do is called Churn Prediction. Unfortunately, the dataset you have is not enough to train a model. You need a variety of different features for a proper prediction model. For ...
Tasos's user avatar
  • 3,910
3 votes
Accepted

Normal distribution instead of Logistic distribution for classification

Nice comparison. Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable ...
Esmailian's user avatar
  • 9,252
3 votes

how to check the distribution of the training set and testing set are similar

Two common scores to quantify the (dis)similarity of distributions are the Kullback-Leibler divergence and the Jensen-Shannon divergence.
pcko1's user avatar
  • 3,930
3 votes
Accepted

A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation

Again, the entropy equation coded was this: entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above. They are the same after ...
Yuq Wang's user avatar
3 votes

How this visualisation was made?

With Wolfram Language you may use "Icon" Entity and ConstantArray to create lists ...
Edmund's user avatar
  • 695
3 votes

Confidence value for face recognition

If I had to calculate such a function, I would: Calculate the probability (not Z-score) for $x$ in a two-tailed test (https://en.wikipedia.org/wiki/One-_and_two-tailed_tests) for the probability that ...
Pieter21's user avatar
  • 1,041
3 votes
Accepted

What is the next after finding best distribution for my data?

Imho there's probably nothing to do with this information, especially considering only the technical side of it. It's highly subjective, but I think that what people mean when they say to "know ...
Erwan's user avatar
  • 25.2k
2 votes

How to check if a data is in gaussian distribution in R or excel?

The Normal distribution is the same as the Gaussian distribution. Its just two names for the same thing. Whatever you do - fit parameters, compute goodness-of-fit, etc - if the documentation says its ...
Spacedman's user avatar
  • 1,982
2 votes

Working with Data which is not Normal/Gaussian

Gaussian models are often used (and maybe sometimes over-used) because of their mathematical convenience (many statistical models can be found as built-in functions, when based on the Gaussian ...
Eskapp's user avatar
  • 446
2 votes
Accepted

Supervised learning for variable length feature-less data

This is a typical scenario in language processing, e.g. if you want to read a product review representing each word with a number and output a 1-5 star rating. You can use a recurrent many-to-one RNN/...
Silpion's user avatar
  • 761
2 votes
Accepted

Does classification of a balanced data-set lead to any problem?

Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't ...
Green Falcon's user avatar
2 votes

can I say that my variable is "approximately" normally distributed?

Many will tell you that normality tests are overly sensitive, especially given that most statistical tests are robust to even gross departures from normality. If you are very concerned, conduct a ...
HEITZ's user avatar
  • 911
2 votes

can I say that my variable is "approximately" normally distributed?

Your data is already binned. You should do a Chi-Squared test to see if the hypothesis that your data is normally distributed is correct. You can refer to this question to see how you can go about ...
JahKnows's user avatar
  • 8,846
2 votes

Clustering by Distributions of Groups of observations

You say "I don't want to calculate slopes or averages by groups and cluster because the distributions don't seem linear, or normal" but it looks to me like each group makes a nice compact cluster ...
G5W's user avatar
  • 303

Only top scored, non community-wiki answers of a minimum length are eligible