7

There are models that do not make assumption that the underlying data distribution is a normal distribution. For example, support vector machine just cares about the boundaries of the separating hyperplane and do not assume the exact shape of the distributions. Decision tree models also do not make such assumption. Gaussian distribution is popular and it ...


7

It looks like the person that wrote the blog is combining the samples from the test set and train set into one dataframe and then predicting if each sample comes from the test set or training set (his y variable is called “is_train” which indicates whether the sample came from the training set or not). I think his point is that if you are able to accurately ...


6

There isn't any built-in function to do this directly in pandas, but by getting the array collection of AxesSubplot, iterating on them to retrieve the matplotlib patches you can achieve the desired result. Here's some dummy data to play with: import pandas as pd import numpy as np df = pd.DataFrame(np.random.randint(low=0, high=3, size=(1000,16))) Now, ...


5

I am a big fan of violin-plots. Although both aim for the same goal (visualizing distributions and key figures for that), box-plots have their limitations. Please have a look into following gif [1]): Box-plots are not able to capture the change in the raw data, while voilin-plots do so: [1] taken from https://www.autodesk.com/research/publications/same-...


5

In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate estimates, meaning that values which are below zero or above one would be rare (at least the problem should be less severe than with OLS). In this case you could simply restrict results to $\hat{y} \...


4

The first returns a probability density of the distributions. As you can see, they integrate to 1, i.e. they cover the same area (because they are probabilities, not the raw data). The second returns actual frequencies, and that's why you have the actual scale of the data. Different histograms having different scales.


4

You could use the reg:logistic objective function. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters Edit: You need to use either the internal API for this, or use the sklearn API's XGBRegressor with objective='reg:logistic' (or binary:logistic). Do not use XGBClassifier, as it will label-encode the target!


4

My go to library would be matplotlib, with which it is relatively easy to generate something similar. I don't have the correct font family to render the exact output as above, but this hopefully illustrates the point Source Code import pandas as pd import numpy as np import matplotlib.pyplot as plt # Create the data to plot on # create a 2d array of ...


4

Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions. Some takeways: Add count of values of features as new features. This is information that LightGBM can’t see Check unique value counts for all features of train and test set. Model stacking almost always works ...


3

Your confusion is apt. Normally distributed data doesn't show up that often. Most of the real world datasets are more complex than normal. Many of natural occurring phenomenon (think: height of people in certain population) might be normal. But most of the cases where human behaviour plays a strong role (donations as you mentioned, incomes, peoples ...


3

Binning One easy way to do such an estimate is to put the continuous values into bins and obtain a discrete problem. Split up the domains of $X$ and $Y$ into bins and count the number of points that fall within each bin to obtain a density. So, the calculation would be: $$ \sum_{b_x \in Bins_x} \sum_{b_y \in Bins_y} \frac {\#(b_x, b_y)} N \log ...


3

Hugely dependent on both user and audience preference (violin plots being more unusual could throw people), so it is mostly up to you. One main reason to go for a violin plot is to give more detail about the distribution, as box plots just give hard stops at the mean, stddev and 2 stddevs. Therefore if you think there is interesting information contained in ...


3

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor. Here are simulated sample data based on normal distribution with three different means similar to your example. By calculating the difference to the previous ...


3

What you are trying to do is called Churn Prediction. Unfortunately, the dataset you have is not enough to train a model. You need a variety of different features for a proper prediction model. For example, you need demographic data for the customers, but most important, data related to customer's actions. For example, if you have a mobile network company, ...


3

I want to view a specific image or a dataset's distribution, and see if they are different. Does this do the trick? It depends what you want to understand or learn about your data. what does each axis mean then? In all of your plots, the x-axis ranges from 0-255, which is because in all your plots, you are creating histograms of the individual pixel ...


3

Nice comparison. Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable derivation consisting of the notorious erf function. Let's first pinpoint what is $x$ in the context of logistic regression. Logistic regression model can be ...


3

Again, the entropy equation coded was this: entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above. They are the same after simple algebraic manipulations. The entropy of a single variable Gaussian distribution with pdf $p(x|\mu, \sigma)$ is \begin{align} \mathcal{H}(p) & = ln(\...


3

With Wolfram Language you may use "Icon" Entity and ConstantArray to create lists of "Crayola" ColorData colored icons and display with Multicolumn 24 columns wide. palette = <|"SeaGreen" -> 135, "Razzmatazz" -> 146, "Yellow" -> 18, "TurquoiseBlue" -> 13|>; Multicolumn[ Flatten@ ...


3

If I had to calculate such a function, I would: Calculate the probability (not Z-score) for $x$ in a two-tailed test (https://en.wikipedia.org/wiki/One-_and_two-tailed_tests) for the probability that $x$ belongs to the distribution for both distributions. Then you have $p_s(x)$ and $p_c(x)$ for self-match/cross-match. Don't use probability $0.1\%$ to reject,...


2

The Normal distribution is the same as the Gaussian distribution. Its just two names for the same thing. Whatever you do - fit parameters, compute goodness-of-fit, etc - if the documentation says its for a Normal distribution then you can say "Gaussian" instead. Completely and totally identical.


2

This is a typical scenario in language processing, e.g. if you want to read a product review representing each word with a number and output a 1-5 star rating. You can use a recurrent many-to-one RNN/GRU/LSTM network: After feeding each word it generates a feature vector which can be fed into a classifier. https://en.wikipedia.org/wiki/...


2

Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't work well because of changing the distribution of each class. In cases like a disease prediction where the number of each class varies for different places, a ...


2

Many will tell you that normality tests are overly sensitive, especially given that most statistical tests are robust to even gross departures from normality. If you are very concerned, conduct a parametric test anyway along with a nonparametric, and if the end results agree, stick with the parametric.


2

Your data is already binned. You should do a Chi-Squared test to see if the hypothesis that your data is normally distributed is correct. You can refer to this question to see how you can go about this. You will set your baseline as the Normal distribution with the mean and variance from your dataset. Then you will test the difference in expected value ...


2

You say "I don't want to calculate slopes or averages by groups and cluster because the distributions don't seem linear, or normal" but it looks to me like each group makes a nice compact cluster which would be represented well by the centroid of the group. Of course, you have 5400 groups and I am only looking at four of them in your sample. Maybe the other ...


2

To visualize change in the size of multiple entities that are contributing to a total through time, e.g. total (t) = book_1 (t) + book_2 (t) + ..., we can use Stacked Area Plot. This plot can be used for normalized and un-normalized (absolute) values. Preprocessing For large number of entities, to avoid cognitive load, we can keep only those entities that ...


2

Two common scores to quantify the (dis)similarity of distributions are the Kullback-Leibler divergence and the Jensen-Shannon divergence.


2

Tried a few things in the OP comments that didn't work. However, you may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for Python you can use these functions in Python. import datetime from wolframclient.evaluation import WolframLanguageSession from wolframclient.language import ...


2

My understanding of your problem: you have realizations of $X$ (property value at first point in time) and $Y$ (value at second point in time). Want to fit a distribution for $Y-X$--more generally, a "model for the data." One way of doing this would be plot a histogram of these values and then come up with plausible distributions that might fit the data, ...


2

You can think of the data distribution as the world that the model lives in. You want to train it to perform well in this world and to do this you have to train it on examples representing this world. Further you have to estimate its performance in this world by testing it on more examples from this world. If you train your model in one world, test and ...


Only top voted, non community-wiki answers of a minimum length are eligible