New answers tagged statistics
1
Simplest approach
First a very fast and perhaps practical approach: just remove them with replacing them!
From your bar chart, it seems you have a lot of transactions - several hundred thousand. Removing a few hundred (I can't even see a bar for the > $600 transactions) and not replacing them wouldn't mean the remaining data is unusable. Replacing those ...
0
You can use any of these below for replacing the outliers
Quantile-based Flooring and Capping
In this technique, we will do the flooring (e.g., the 10th percentile) for the lower values and capping (e.g., the 90th percentile) for the higher values. The lines of code below print the 10th and 90th percentiles of the variable 'amount', respectively. These ...
2
Performance Measures for Multi-Class Problems
For classification problems, classifier performance is typically defined according to the confusion matrix associated with the classifier. Based on the entries of the matrix, it is possible to compute sensitivity (recall), specificity, and precision. For a single cutoff, these quantities lead to balanced accuracy ...
2
When I see it correctly in the top figure, there is some " bunching" in your data, meaning that there are a number of companies with the same (or very similar) number of employees. Since you appear to run a regression with only one independent variable (right hand side), this bunching will be visible in the residual.
Your model is:
$$ y_i = \beta_0 ...
0
Your whole modeling framework is inefficient because a large fraction of observations cluster in a small fraction of the range. You would benefit from switching to log(Profit) and log(Number of Employees). Then the effects you are seeing will be less pronounced. Then the fitted distribution of the small companies will not be as influenced by the long tail of ...
2
To help you find other resources, this is commonly referred to as "Missing Not At Random."
Some models, like xgboost, handle missing values inherently, making tree splits at a real value but then choosing which branch to send the missing values along. (Other implementations of CART don't do that, and the Quinlan family of trees does something ...
2
The common case of missing values for which data is imputed or removed assumes that missing values appear randomly in the data, so the absence of value has no relevance to the task.
From your description, in your data the fact that a value is missing is significant by itself. So I'd say that yes, it makes sense in this case to represent this information as a ...
0
This is an active area of research called Human Activity Recognition. There are several public datasets available to cross-validate your methods, and you might want to start here: UCI HAR Dataset. There's a paper that accompanies the dataset that describes their preprocessing methods, so you'll want to have a look at that and see if anything helps in terms ...
0
I think these are often used colloquially as synonyms, but let's try to find the differences.
Each of them begins with "Time Series" (TS). So the difference lies in the three following terms. here with my interpretation:
Analysis - wanting to describe and understand characteristics the observed data coming from the generating function$^1$.
...
0
Accuracy is function of data quality. If the labels are quite similar and you have less data then it's alright for initial phase. To further evaluate your results you could find the prediction results per class using classifier report and see which label class is least performing and then look for better image/text quality or preprocessing to improve the ...
0
The current approach use 70/30 or 80/20, the most used is 80/20 (train/test). However there is other things you should check, for example if you data is balanced. If your data is not balanced you might want to use undersample or oversample.
0
The bias of an estimator $\hat{f}(x)$ of $f(x)$ is by definition:
$$Bias(\hat{f},f)(x)=E[\hat{f}(x)]-f(x)$$
which is some definite (not-random) function $\phi(x)$.
Why this is so? The intuitive way to understand it is that estimation itself is a random process. Given random data will produce a random estimation. However the expected estimation is $E[\hat{f}(...
0
It seems it is even easier
The MLE is defined as
$$ \theta_{MLE} = \arg\max -(100 \ln \pi + \sum_{i=1}^{100} \ln(1 + (x_{i} - \theta)^{2})) $$
so you need to minimize the sum of logs and applying the exponential to each element of the sum does not change the result of the argmin because it is a monotone increasing function, so at the end of the day you have ...
Top 50 recent answers are included
