9
votes
Accepted
Plotting different values in pandas histogram with different colors
There isn't any built-in function to do this directly in pandas, but by getting the array collection of AxesSubplot, iterating on them to retrieve the matplotlib ...
- 270
8
votes
Accepted
Working with Data which is not Normal/Gaussian
There are models that do not make assumption that the underlying data distribution is a normal distribution.
For example, support vector machine just cares about the boundaries of the separating ...
- 3,128
8
votes
Accepted
how to check the distribution of the training set and testing set are similar
It looks like the person that wrote the blog is combining the samples from the test set and train set into one dataframe and then predicting if each sample comes from the test set or training set (his ...
- 825
7
votes
Boxplots or violinplots?
I am a big fan of violin-plots. Although both aim for the same goal (visualizing distributions and key figures for that), box-plots have their limitations. Please have a look into following gif 1): ...
- 171
7
votes
xgboost: Is there a way to perform regression on rates/percentages data?
In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate ...
- 7,776
6
votes
xgboost: Is there a way to perform regression on rates/percentages data?
You could use the reg:logistic objective function. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters
Edit: You need to use either ...
- 12.3k
6
votes
Why do train, test, validation datasets need to have the same distribution?
You can think of the data distribution as the world that the model lives in. You want to train it to perform well in this world and to do this you have to train it on examples representing this world. ...
- 1,555
6
votes
What is the difference between Covariate Shift, Label Shift, Concept Shift, Concept Drift, and Prior Probability Shift?
I am paraphrasing from the book "Designing Machine Learning Systems by Chip Huyen".
In a supervised machine learning problem, the training dataset can be viewed as a set of samples from a ...
- 161
5
votes
Accepted
How can I plot/display a dataset or an image distribution?
I want to view a specific image or a dataset's distribution, and see
if they are different. Does this do the trick?
It depends what you want to understand or learn about your data.
what does ...
- 15.1k
4
votes
Which outlier detection can detect these outliers?
You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor.
...
- 1,113
4
votes
Accepted
Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?
The first returns a probability density of the distributions. As you can see, they integrate to 1, i.e. they cover the same area (because they are probabilities, not the raw data).
The second returns ...
- 6,355
4
votes
How this visualisation was made?
My go to library would be matplotlib, with which it is relatively easy to generate something similar.
I don't have the correct font family to render the exact output as above, but this hopefully ...
- 431
4
votes
Regression: How to deal with positive skewness in continuous target variable
Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions.
Some takeways:
Add count of values of features as ...
- 5,759
3
votes
How to predict whether or not a customer will renew
What you are trying to do is called Churn Prediction. Unfortunately, the dataset you have is not enough to train a model. You need a variety of different features for a proper prediction model.
For ...
- 3,940
3
votes
Boxplots or violinplots?
Hugely dependent on both user and audience preference (violin plots being more unusual could throw people), so it is mostly up to you.
One main reason to go for a violin plot is to give more detail ...
- 393
3
votes
How often do we see normally distributed data
Your confusion is apt. Normally distributed data doesn't show up that often. Most of the real world datasets are more complex than normal. Many of natural occurring phenomenon (think: height of people ...
- 2,018
3
votes
Accepted
Normal distribution instead of Logistic distribution for classification
Nice comparison.
Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable ...
- 9,442
3
votes
how to check the distribution of the training set and testing set are similar
Two common scores to quantify the (dis)similarity of distributions are the Kullback-Leibler divergence and the Jensen-Shannon divergence.
- 3,970
3
votes
Accepted
A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation
Again, the entropy equation coded was this: entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above.
They are the same after ...
- 46
3
votes
How this visualisation was made?
With Wolfram Language you may use "Icon" Entity and ConstantArray to create lists ...
- 745
3
votes
Confidence value for face recognition
If I had to calculate such a function, I would:
Calculate the probability (not Z-score) for $x$ in a two-tailed test (https://en.wikipedia.org/wiki/One-_and_two-tailed_tests) for the probability that ...
- 1,051
3
votes
Accepted
What is the next after finding best distribution for my data?
Imho there's probably nothing to do with this information, especially considering only the technical side of it.
It's highly subjective, but I think that what people mean when they say to "know ...
- 25.9k
2
votes
How to check if a data is in gaussian distribution in R or excel?
The Normal distribution is the same as the Gaussian distribution. Its just two names for the same thing. Whatever you do - fit parameters, compute goodness-of-fit, etc - if the documentation says its ...
- 1,992
2
votes
Accepted
Supervised learning for variable length feature-less data
This is a typical scenario in language processing, e.g. if you want to read a product review representing each word with a number and output a 1-5 star rating.
You can use a recurrent many-to-one RNN/...
- 761
2
votes
Accepted
Does classification of a balanced data-set lead to any problem?
Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't ...
- 14.2k
2
votes
can I say that my variable is "approximately" normally distributed?
Many will tell you that normality tests are overly sensitive, especially given that most statistical tests are robust to even gross departures from normality. If you are very concerned, conduct a ...
- 911
2
votes
can I say that my variable is "approximately" normally distributed?
Your data is already binned. You should do a Chi-Squared test to see if the hypothesis that your data is normally distributed is correct. You can refer to this question to see how you can go about ...
- 9,056
2
votes
Clustering by Distributions of Groups of observations
You say "I don't want to calculate slopes or averages by groups and cluster because the distributions don't seem linear, or normal" but it looks to me like each group makes a nice compact cluster ...
- 303
2
votes
Working with Data which is not Normal/Gaussian
Gaussian models are often used (and maybe sometimes over-used) because of their mathematical convenience (many statistical models can be found as built-in functions, when based on the Gaussian ...
- 456
2
votes
How to visualize change in a distribution with a few outliers that account for a very large percent of the total?
To visualize change in the size of multiple entities that are contributing to a total through time, e.g. total (t) = book_1 (t) + book_2 (t) + ..., we can use Stacked Area Plot. This plot can be used ...
- 9,442
Only top scored, non community-wiki answers of a minimum length are eligible