# Tag Info

13

It depends what question you are trying to answer. You are looking at the rate of change of a time series, and it sounds like you are trying to show how that changed over time. The mean gives the reader one intuitive insight: they can trivially estimate the number of followers at any date $d$ days since the start by multiplying by the mean rate of change. ...

10

The arithmetic mean is denoted as $\bar{x}$ $$\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$$ where each $x_i$ represent an unique observation. The arithmetic mean measures the average value for a given set of numbers. In contrast to this, the median is the value which falls directly in the middle of your dataset. The median is especially useful when you are ...

5

The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is from GDELT, which provides this interval for average tone of news documents. Asking if equally distancing the emotions is statistically correct does not make ...

3

The first question is: what to do you want to see in user profile? Top-10 tracks, top-10 artist by user? How many tracks/artists a user listens to in a day on average (may be, in last month)? May be you want to get some general information related to the whole user base: Which artist/track is the most popular among users from different countries (top-N of ...

3

You could think of histogram as one way of plotting a distribution of values. Another way of plotting such distribution is a KDE (Kernel Density Estimate), but after all, these inform about the same concept, which is how frequent is each value (or values interval) of your values series (x-axis). I like this picture from seaborn distplot to have a one shot ...

2

As you mentioned their goals are different. In clustering, we try to group data such that they have the same variability. For example, clustering customers of a company into different clusters, somehow, members of each cluster have the same behavior in their buying. On the other side, in blocking we try to reduce the variability, to record linkage, as an ...

2

Simply to say, If your data is corrupted with noise or say erroneous no.of twitter followers as in your case, Taking mean as a metric could be detrimental as the model will perform badly. In this case, If you take the median of the values, It will take care of outliers in the data. Hope it helps

2

I think there is no answer to your question since there is no absolute universal "good". Everything depends on the question you ask and the tools you use. This is why there are a lot of imputation techniques. There is no replacement for a missing value. However, in the constrains given by your question and used tools, you can think of imputation which does ...

2

You can download a free as in beer software Qlikview that allows you to do interactive data discovery via graphical interface similar to Excel but also featuring a powerful scripting language for data load and transformation. Huge flat files is no problem at all. It is an in-memory technology so you'd need a computer with a lot of RAM. The advantage though ...

2

A simple and direct answer is that skewness and kurtosis are both defined in terms of the $Z-$values, $Z = (X-\mu)/\sigma$ : Skewness = $E(Z^3)$ and Kurtosis = $E(Z^4)$. When talking about a data set, you can just replace the expectation operator "$E$" with the ordinary average. Since raising a number to either the third or fourth power amplifies values ...

2

You can use class ratio / sample class ratio. Which will make it more intuitive for any reader while going through the details. As its not used for model performance analysis hence I think we don’t have a metric name for this.

2

Good question. Lots of options. Most would recommend you use CDFs instead of histograms. So convert your observed distribution to empirical CDF (in R it’s ecdf() - I dint know python). Then, plot another line that is the theoretical CDF using your best-fit parameters over the span of a to b, where a to b gives you at least the same coverage as your ...

2

Cochran's Q test Is a generalisation of the McNemars test and can be used to see if there is a truly better classifier for the metric chosen. You can ofcourse also do pairwise Mcnemare test and draw conclusions from there. NOTE: These things are expensive

2

First figure shows Frequency(Y-axis) distribution over varied values of Line data(X-axis). Similar information gets conveyed by your second figure as well, but second one provides a deeper insight to frequency fluctuation over smaller bins of Line data. Additionally in second figure, various types (Lognorm, Exponential, etc.) of distribution gets line traced ...

1

It's similar to the concept of odds. The baseline count of positive and negative samples gives you an odds (all else being equal) that a random sample from the population will be positive or negative.

1

This is Imbalance Class Dataset concept. Mostly the ratio you mentioned is used as: # of positive class (minority) / # of negative class (majority) For example: "The dataset contains 100 fraud activities among 10000 transactions with 0.01 class imbalance." There are no strict rule/metric about that you can also use other version with 100 class imbalance ...

1

It can be a good idea, but if you want to use PCA, you will have to use it carefully. First of all, PCA will reduce dimension depending on the data observed in your dataset. Consequently, if it is biaised somehow, the projection will not work with different datasets. For instance, if you have a strong correlation between two features and a third one is ...

1

PCA will change your data and you will not be able to interpret it in sane sense, you can just slice and dice the data and do many things by hand, PCA would be usefull if you would want to find "neighbouring" players in terms of raw statistics but it can be deceptive because PCA don't know which stats are important, if you want to decrease dimensionality of ...

1

I see. Yes PCA is a good tool that you can use on your choice of attributes. The tricky part is that PCA is based on variance. Hence, if you have 2 attributes which are more important and have low variance and then 5 more with a lot of variances then the latter will be predominant on the first PC (I guess some standardisation would help). Hence, maybe it ...

1

The normal distribution is a theoretical model of data. Empirical data can be distributed more similarily or more dissimilarly to a normal distribution. That empirical data has a couple of notable divergences from a theoretical normal distribution: Presence of outliers Not symmetric Relative "peakedness" Depending on your goal, you can pick a better model ...

1

Ok, I am not sure if I understood it correctly. But if it is only the right-side figure equation it would be: =1/(1+EXP(3.1058)) (I have tested on LibreOffice only, but it should work on excel as well) Now, if you want to do something more complex using the prior knowledge of event A, it should be more clear what is exactly this event and how it is ...

1

I find myself explaining this a lot and the example I use is the famous Bill Gates version. Bill Gates is in your data science class. Your instructor asks you: what is the average income or net worth of this class? Bill Gates sheepishly obliges and tells you what his income is. Now when you say the average income of your group is a zillion dollars - ...

1

Often median is more robust to extreme value to mean. Try to think it as a minimization task. Median corresponds to absolute loss while mean corresponds to square loss.

1

1

What you need to do is called One Hot Encoding. There are two ways to do. One is using Scikit-learn as described in Scikit-Learn documentation or use get_dummies from pandas. Example 1: from sklearn.preprocessing import OneHotEncoder status_encoder = OneHotEncoder() city_encoder = OneHotEncoder() X = status_encoder.fit_transform(df.status.values.reshape(-1,...

1

How about implementing one of the most basic binary classifiers by hand without any libraries, Logistic Regression? Here is the explanation of the Logistic Regression in subtle terms: https://www.youtube.com/watch?v=6tByJTacCOc&index=6&list=PLBAGcD3siRDguyYYzhVwZ3tLvOyyG5k6K Here is the description of writing a simplified version of a binary ...

1

As the comments suggest, it's not always helpful to think of points outside the whiskers as "outliers". What you are seeing in these boxplots is a strong positive skew. And yes, a strongly skewed target is typically harder to predict than a less-skewed target. In traditional survival analysis, it's common to model this skewed data using a parametric ...

1

The normalization factor is used to reduce any probability function to a probability density function with total probability of one. See Wikipedia. Say your unnormalized value is [0.1, 0.2, 0.3, 0.2]. You normalize it by dividing it by $z_t=0.1+0.2+0.3+0.2=0.8$, therefore get the normalized value [0.125, 0.25, 0.375, 0.25] which sums up to 1..

1

Perhaps you can define an "active user" (e.g., someone who has made a purchase in the last 3 months, etc. Choose a period sensible for the typical purchase cycle) and calculate the something along the line of DAU or MAU, etc. Say, if a typical active user will on average buy something from your store every 3 months, assume a user has churned if she fails to ...

1

Just to clarify the question: Do the lists describe different graphs, or do you need the similarities to learn which lists refer to the same graph? Is data tri- or duplicated within single lists or are lists duplicating other lists? Would you consider removing redundant metrics? Sorry, this was rather a comment than an answer, but I am not yet entitled to ...

Only top voted, non community-wiki answers of a minimum length are eligible