# Tag Info

13

It depends what question you are trying to answer. You are looking at the rate of change of a time series, and it sounds like you are trying to show how that changed over time. The mean gives the reader one intuitive insight: they can trivially estimate the number of followers at any date $d$ days since the start by multiplying by the mean rate of change. ...

10

The arithmetic mean is denoted as $\bar{x}$ $$\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$$ where each $x_i$ represent an unique observation. The arithmetic mean measures the average value for a given set of numbers. In contrast to this, the median is the value which falls directly in the middle of your dataset. The median is especially useful when you are ...

6

No. What you refer to (the difference between the means of group A and B) is actually the effect size, and it has absolutely nothing to do with the p-values. The situation is nicely summarized in the (highly recommended) paper Using Effect Size—or Why the P Value Is Not Enough (emphasis mine): Why Report Effect Sizes? The effect size is the main finding of ...

5

The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is from GDELT, which provides this interval for average tone of news documents. Asking if equally distancing the emotions is statistically correct does not make ...

4

Comparative study of Mutual Information measures A comparative study of standard Shannon entropy along with variations of Coding Theorem Method(CTM) and Block Decomposition Model(BTM) as shown in A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity General guide for Estimating mutual information and ...

3

The first question is: what to do you want to see in user profile? Top-10 tracks, top-10 artist by user? How many tracks/artists a user listens to in a day on average (may be, in last month)? May be you want to get some general information related to the whole user base: Which artist/track is the most popular among users from different countries (top-N of ...

3

You could think of histogram as one way of plotting a distribution of values. Another way of plotting such distribution is a KDE (Kernel Density Estimate), but after all, these inform about the same concept, which is how frequent is each value (or values interval) of your values series (x-axis). I like this picture from seaborn distplot to have a one shot ...

2

You can download a free as in beer software Qlikview that allows you to do interactive data discovery via graphical interface similar to Excel but also featuring a powerful scripting language for data load and transformation. Huge flat files is no problem at all. It is an in-memory technology so you'd need a computer with a lot of RAM. The advantage though ...

2

I think there is no answer to your question since there is no absolute universal "good". Everything depends on the question you ask and the tools you use. This is why there are a lot of imputation techniques. There is no replacement for a missing value. However, in the constrains given by your question and used tools, you can think of imputation which does ...

2

As you mentioned their goals are different. In clustering, we try to group data such that they have the same variability. For example, clustering customers of a company into different clusters, somehow, members of each cluster have the same behavior in their buying. On the other side, in blocking we try to reduce the variability, to record linkage, as an ...

2

A simple and direct answer is that skewness and kurtosis are both defined in terms of the $Z-$values, $Z = (X-\mu)/\sigma$ : Skewness = $E(Z^3)$ and Kurtosis = $E(Z^4)$. When talking about a data set, you can just replace the expectation operator "$E$" with the ordinary average. Since raising a number to either the third or fourth power amplifies values ...

2

Simply to say, If your data is corrupted with noise or say erroneous no.of twitter followers as in your case, Taking mean as a metric could be detrimental as the model will perform badly. In this case, If you take the median of the values, It will take care of outliers in the data. Hope it helps

2

You can use class ratio / sample class ratio. Which will make it more intuitive for any reader while going through the details. As its not used for model performance analysis hence I think we don’t have a metric name for this.

2

Good question. Lots of options. Most would recommend you use CDFs instead of histograms. So convert your observed distribution to empirical CDF (in R it’s ecdf() - I dint know python). Then, plot another line that is the theoretical CDF using your best-fit parameters over the span of a to b, where a to b gives you at least the same coverage as your ...

2

Cochran's Q test Is a generalisation of the McNemars test and can be used to see if there is a truly better classifier for the metric chosen. You can ofcourse also do pairwise Mcnemare test and draw conclusions from there. NOTE: These things are expensive

2

First figure shows Frequency(Y-axis) distribution over varied values of Line data(X-axis). Similar information gets conveyed by your second figure as well, but second one provides a deeper insight to frequency fluctuation over smaller bins of Line data. Additionally in second figure, various types (Lognorm, Exponential, etc.) of distribution gets line traced ...

2

According to The Search for Categorical Correlation post on TowardsDataScience, one can use a variation of correlation called Cramer's association. Going categorical What we need is something that will look like correlation, but will work with categorical values — or more formally, we’re looking for a measure of association between two categorical features. ...

1

This is Imbalance Class Dataset concept. Mostly the ratio you mentioned is used as: # of positive class (minority) / # of negative class (majority) For example: "The dataset contains 100 fraud activities among 10000 transactions with 0.01 class imbalance." There are no strict rule/metric about that you can also use other version with 100 class imbalance ...

1

It can be a good idea, but if you want to use PCA, you will have to use it carefully. First of all, PCA will reduce dimension depending on the data observed in your dataset. Consequently, if it is biaised somehow, the projection will not work with different datasets. For instance, if you have a strong correlation between two features and a third one is ...

1

PCA will change your data and you will not be able to interpret it in sane sense, you can just slice and dice the data and do many things by hand, PCA would be usefull if you would want to find "neighbouring" players in terms of raw statistics but it can be deceptive because PCA don't know which stats are important, if you want to decrease dimensionality of ...

1

I see. Yes PCA is a good tool that you can use on your choice of attributes. The tricky part is that PCA is based on variance. Hence, if you have 2 attributes which are more important and have low variance and then 5 more with a lot of variances then the latter will be predominant on the first PC (I guess some standardisation would help). Hence, maybe it ...

1

The normal distribution is a theoretical model of data. Empirical data can be distributed more similarily or more dissimilarly to a normal distribution. That empirical data has a couple of notable divergences from a theoretical normal distribution: Presence of outliers Not symmetric Relative "peakedness" Depending on your goal, you can pick a better model ...

1

Ok, I am not sure if I understood it correctly. But if it is only the right-side figure equation it would be: =1/(1+EXP(3.1058)) (I have tested on LibreOffice only, but it should work on excel as well) Now, if you want to do something more complex using the prior knowledge of event A, it should be more clear what is exactly this event and how it is ...

1

I find myself explaining this a lot and the example I use is the famous Bill Gates version. Bill Gates is in your data science class. Your instructor asks you: what is the average income or net worth of this class? Bill Gates sheepishly obliges and tells you what his income is. Now when you say the average income of your group is a zillion dollars - ...

1

Often median is more robust to extreme value to mean. Try to think it as a minimization task. Median corresponds to absolute loss while mean corresponds to square loss.

1

You're right that it is a predictive model, but analysing feature importance, partial dependence and other metrics/plots are intended to allow you to gain an understanding of the classification process of models which are often very black-box in nature. Yes, you can use this to gain some understanding of the underlying data, but I'd be more inclined to say ...

1

It can be done like: import pandas as pd melted = pd.melt(df1, id_vars=["index"], var_name="Var", value_name="Score").dropna() table=pd.crosstab(index=melted['index'], columns=melted['Score']) %matplotlib inline table.plot.bar() #for simple axes subplots

1

Clustering is definitely something that can help. As you describe, the issue is that you can see that not all instances "behave" the same way. So if you can cluster them into groups that do behave more similarly, you would probably improve. Any clustering technique would work in theory. Although, techniques like K-means for instance do favour clusters of ...

1