8 votes
Accepted

Plotting different values in pandas histogram with different colors

There isn't any built-in function to do this directly in pandas, but by getting the array collection of AxesSubplot, iterating on them to retrieve the matplotlib ...
8 votes
Accepted

Working with Data which is not Normal/Gaussian

There are models that do not make assumption that the underlying data distribution is a normal distribution. For example, support vector machine just cares about the boundaries of the separating ...
7 votes
Accepted

how to check the distribution of the training set and testing set are similar

It looks like the person that wrote the blog is combining the samples from the test set and train set into one dataframe and then predicting if each sample comes from the test set or training set (his ...
7 votes

xgboost: Is there a way to perform regression on rates/percentages data?

In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate ...
  • 6,962
5 votes

Boxplots or violinplots?

I am a big fan of violin-plots. Although both aim for the same goal (visualizing distributions and key figures for that), box-plots have their limitations. Please have a look into following gif [1]): ...
4 votes

Which outlier detection can detect these outliers?

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor. ...
4 votes
Accepted

Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?

The first returns a probability density of the distributions. As you can see, they integrate to 1, i.e. they cover the same area (because they are probabilities, not the raw data). The second returns ...
  • 5,697
4 votes

xgboost: Is there a way to perform regression on rates/percentages data?

You could use the reg:logistic objective function. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters Edit: You need to use either ...
  • 10.1k
4 votes

How this visualisation was made?

My go to library would be matplotlib, with which it is relatively easy to generate something similar. I don't have the correct font family to render the exact output as above, but this hopefully ...
4 votes

Regression: How to deal with positive skewness in continuous target variable

Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions. Some takeways: Add count of values of features as ...
  • 5,369
3 votes

How often do we see normally distributed data

Your confusion is apt. Normally distributed data doesn't show up that often. Most of the real world datasets are more complex than normal. Many of natural occurring phenomenon (think: height of people ...
  • 1,938
3 votes

How to estimate the mutual information numerically?

Binning One easy way to do such an estimate is to put the continuous values into bins and obtain a discrete problem. Split up the domains of $X$ and $Y$ into bins and count the number of points that ...
  • 286
3 votes

Boxplots or violinplots?

Hugely dependent on both user and audience preference (violin plots being more unusual could throw people), so it is mostly up to you. One main reason to go for a violin plot is to give more detail ...
  • 383
3 votes

How to predict whether or not a customer will renew

What you are trying to do is called Churn Prediction. Unfortunately, the dataset you have is not enough to train a model. You need a variety of different features for a proper prediction model. For ...
  • 3,770
3 votes
Accepted

How can I plot/display a dataset or an image distribution?

I want to view a specific image or a dataset's distribution, and see if they are different. Does this do the trick? It depends what you want to understand or learn about your data. what does ...
  • 14.1k
3 votes
Accepted

Normal distribution instead of Logistic distribution for classification

Nice comparison. Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable ...
  • 8,667
3 votes
Accepted

A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation

Again, the entropy equation coded was this: entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above. They are the same after ...
3 votes

How this visualisation was made?

With Wolfram Language you may use "Icon" Entity and ConstantArray to create lists ...
  • 695
3 votes

Confidence value for face recognition

If I had to calculate such a function, I would: Calculate the probability (not Z-score) for $x$ in a two-tailed test (https://en.wikipedia.org/wiki/One-_and_two-tailed_tests) for the probability that ...
  • 1,021
2 votes

How to check if a data is in gaussian distribution in R or excel?

The Normal distribution is the same as the Gaussian distribution. Its just two names for the same thing. Whatever you do - fit parameters, compute goodness-of-fit, etc - if the documentation says its ...
  • 1,932
2 votes

Working with Data which is not Normal/Gaussian

Gaussian models are often used (and maybe sometimes over-used) because of their mathematical convenience (many statistical models can be found as built-in functions, when based on the Gaussian ...
  • 436
2 votes
Accepted

Supervised learning for variable length feature-less data

This is a typical scenario in language processing, e.g. if you want to read a product review representing each word with a number and output a 1-5 star rating. You can use a recurrent many-to-one RNN/...
  • 751
2 votes
Accepted

Does classification of a balanced data-set lead to any problem?

Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't ...
  • 13.5k
2 votes

can I say that my variable is "approximately" normally distributed?

Many will tell you that normality tests are overly sensitive, especially given that most statistical tests are robust to even gross departures from normality. If you are very concerned, conduct a ...
  • 901
2 votes

can I say that my variable is "approximately" normally distributed?

Your data is already binned. You should do a Chi-Squared test to see if the hypothesis that your data is normally distributed is correct. You can refer to this question to see how you can go about ...
  • 8,548
2 votes

Clustering by Distributions of Groups of observations

You say "I don't want to calculate slopes or averages by groups and cluster because the distributions don't seem linear, or normal" but it looks to me like each group makes a nice compact cluster ...
  • 293
2 votes

How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

To visualize change in the size of multiple entities that are contributing to a total through time, e.g. total (t) = book_1 (t) + book_2 (t) + ..., we can use Stacked Area Plot. This plot can be used ...
  • 8,667
2 votes

how to check the distribution of the training set and testing set are similar

Two common scores to quantify the (dis)similarity of distributions are the Kullback-Leibler divergence and the Jensen-Shannon divergence.
  • 3,760
2 votes

Plotting Weibull distribution on Wind Speed

Tried a few things in the OP comments that didn't work. However, you may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for ...
  • 695
2 votes
Accepted

Fitting a distribution to data

My understanding of your problem: you have realizations of $X$ (property value at first point in time) and $Y$ (value at second point in time). Want to fit a distribution for $Y-X$--more generally, a "...

Only top scored, non community-wiki answers of a minimum length are eligible