New answers tagged

0

Yes, the problem of imbalance is indeed genuine while pre processing. There are no hard and fast rules for removing outliers, but generic methodologies (percentile,boxplot,Z-score etc). Like gender, if you take salary of all employess then removing outliers means eliminating all highly paid employees.That will make your model learn more about middle/average ...


0

If you find that the classes are somewhat correlated (e.g. being a politican and a businessman is extremely likely to be leader) then you can make use of these statistical rules to bypass or post-modify the votes. Otherwise, I think the best you can do is set a static threshold which gives you the best precision and recall combination.


0

Removing outliers in a high-dimensional scenario can for example be done after dimension reduction by principal component analysis. In the dimension-reduced space either boxplots (1 dimension), bagplots (2 dimension) or gemplots (3 dimensions) can be applied to detect outliers. For details please look at Kruppa, J., & Jung, K. (2017). Automated ...


0

Traditional clustering algorithm which uses Euclidean based distance fails to yield good results in high dimensional data due to Curse of dimensionality Because mean distance between data points diverges and looses its meaning which in turn leads to the divergence of the Euclidean distance, the most common distance used for clustering. So if you are using ...


0

Can I make a slightly more general observation? Look at the datasets they test on, and read [a]. There is no read evidence that this idea works, so why bother to implement it? [a] Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress https://arxiv.org/abs/2009.13807


0

As you are doing joins, I would suggest you to remove outliers and anything on original weather data itself. As once you join number fo flights on each day may have an impact on it. As in future number of passenger would be a target variable, i would suggest you to seprate your data into training (Rows where Number of missing is present) vs number of missing ...


0

GPS data includes positional and time data. If the n+1 position at t+1 is too far away from the n position at t (i.e. d>0.5m for instance), you should be able to detect an anomaly. Same topic about the angle: if the angle between d1 and d2 is grater than a normal value (ex: 2 degree) then it should be considered as an anomaly. You should consider the ...


0

To a human observer it is obvious that "A" and "E" are different because they show a different pattern in their amplitude compared to "B" "C" and "D". The trajectory of "E" doesn't even look that different for the most part, it just jumps up and down rapidly at certain intervals. My idea would be to ...


0

One method to consider is Dynamic Time Warping (DTW), which measures similarity between time series. DTW is capable of comparing time series of different lengths, and the resulting score could be used to determine which series are most unique in your sample. You'll find in many articles that DTW works well with KNN classification (I personally can attest to ...


0

If you have a labeled dataset $f(X) =Y$ then you have a supervised learning problem, so you may try to solve it as a "usual" binary classification problem by using metrics like $F1$ or $AUC$ and Cross-validation to evaluate your model's performance, and what I mean by usual is that you do not need to apply something special for anomaly detection ...


Top 50 recent answers are included