3
$\begingroup$

For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a person asks why did you choose this combination and algorithms!? You can reach different combinations and use other algorithms and they may yield better results. What should I answer? I cannot just say based on tests as there are many other algorithms that I haven't tested (cannot test all algorithms). It is not a logical response, I think.

I'm looking for tests to justify my selected methods and combination and say why I have selected these methods.

Please let me know your suggestions.

$\endgroup$
  • $\begingroup$ If you use a few different standard methods, cite a good textbook that proposes such methods, and the list of outliers detected is fairly consistent and robust across these few methods, then you have some basis for your claim. $\endgroup$ – Anna SdTC Feb 20 '17 at 22:40
  • 1
    $\begingroup$ You haven't mentioned your model and what you're doing. $\endgroup$ – HelloWorld Feb 20 '17 at 23:13
  • $\begingroup$ On what basis do you think that a statistical outlier is also one that is illegally manipulated? $\endgroup$ – Paul Feb 22 '17 at 20:39
  • $\begingroup$ Based on the context that I work $\endgroup$ – Arkan Feb 22 '17 at 20:42
  • $\begingroup$ Can you elaborate on that? $\endgroup$ – Paul Feb 23 '17 at 15:28
3
$\begingroup$

You can justify your choices by using data.

Treat the anomaly detection like a supervised learning problem where the concept is being anomaly. Then you'll be able to present - for each method - its confusion matrix. Not only it will be a good justification, it will enable to understand the expected results.

Many times, we have models and we wonder which confidence threshold we should use for alerting. In the supervised learning framework you'll be able to do trade-off like "increasing the confidence to X will lead to a better precision Y yet a decrease of the recall to Z".

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you. Yes, I have presented the result and it is acceptable. But the problem is that we cannot test all methods and their combinations as there are plenty of algorithms. Suppose one says you may get better result if replace grubb method with IQR or many other algorithms which lead to different combinations. Now I should say my reasons for my selections and then mention that there is no need to test other methods as our results are acceptable. I'm wondering that is there any solution by inspecting data distribution and relating it to functionality of used methods? $\endgroup$ – Arkan Feb 21 '17 at 10:59
  • 1
    $\begingroup$ It is very common to have more ideas and methods than time to evaluate them. If we continue in the supervised framework, you can turn each method into a feature. Then you can use feature selection to find the methods that works best for you case. You'll be able to omit methods that aren't useful or contribute nothing more than a different method that you already use. After that you will be able to build a model upon these feature in order to find a way to aggregate them. This direction isn't the optimal but it is a common useful tradeoff. $\endgroup$ – DaL Feb 21 '17 at 11:15
  • $\begingroup$ Thank you. How can we turn a method to a feature? Am I right by the following description: We calculate the standard deviation (SD) of the vector and add it as a new feature. Calculate median and add it as a new feature and etc. Then we use feature selection (say CFS method) and if for example SD of the vector remains (is highly correlated with target value) we say SD method is better than median method in this context? Please explain more. $\endgroup$ – Arkan Feb 21 '17 at 12:31
  • 1
    $\begingroup$ You can use the internal parameters of the method as features but it is not required. It is enough to use the predication itself of the method - whether it is considered an anomaly. Let's say that you use the 3 sigma rule(en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule). There is no need the store the mean and standard deviation as features. Just store whether the value is at least 3 SDs above the mean. $\endgroup$ – DaL Feb 21 '17 at 13:23
  • 1
    $\begingroup$ I don't see this answers the question. $\endgroup$ – HelloWorld Feb 22 '17 at 5:56
3
$\begingroup$

I would add to Dan Levin's answer that when you want to justify a method, the "scientific/engineering way" is to first produce a bibliographical study, where you basically prove that your approach covers an important part of what is commonly known as state-of-the-art methods. I would resume this as follow:

  1. Look for commonly used methods that are known to be efficient for outlier detection.
  2. Summarize their applicability domains (medicine, biology, network security..) and try to link their strong points to your application in order to select some promising methods.
  3. Try the selected methods with usual validation processes inherent to machine learning problems.

Defining what the state-of-the-art consists of a lot of work, is absolutely necessary and is very specific to your application.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.