Consistency based model selection does not perform well for many datasets for One-Class Classification (OCC). So I am looking for some other model selection criteria.

Since, only one class of data (i.e. the positive class) is available, classification error is no longer a valid criterion. It is very challenging task to decide on the optimal parameters in OCC. Here, model selection means the optimal parameter selection for the one-class classifier.

Please suggest any additional methods except consistency based model selection method.

  • $\begingroup$ Question: if you or an "expert" were shown an instance would you be able to reliably classify it into the positive or negative class? $\endgroup$ – Brandon Loudermilk May 21 '16 at 12:04

Have you read Janssens' dissertation "Outlier Selection and One-Class Classification"? He has a chapter on evaluation which may be of use. Have you thought about artificial generation of negative instances?

I had to deal with OCC evaluation awhile back and never came upon an entirely satisfactory solution. As I recall, the basic problem boils down to the fact that you don't really have labelled instances of the negative class. Without these instances there is no way of computing precision = TP/(TP+FP) which relies upon the count of incorrectly accepted negative instances (FP). This just leaves you with recall = TP/(TP + FN) as a computable metric - which by itself is next to useless in evaluating classifier performance i.e. it's trivial to write a classifier that get's 100% recall, just predict the positive class for every instance in your test data... you will correctly identify every true positive instance, but you will also incorrectly predict every negative instance as a positive instance. What is typically required is a balance or trade-off between precision and recall.

You didn't mention a particular problem domain in which you are working, but perhaps you can leverage this domain knowledge as a work-around to the problem of OCC evaluation. This particular problem I will relate is technically cast as a PU Learning problem (learning from Positive and Unlabelled instances). I'll try to describe some solutions that I explored while tackling this challenge - I'll give you fair warning that they are not entirely satisfactory solution, but I believe they are logically sound and somewhat defensible. From a purely pragmatic perspective, they are definitely better than the alternative.

Implicit Negative Class

Scenario: imagine a membership-based incentive program, in which people can pay an annual fee to become a member of Foobar Inc. Members, but not non-members, are entitled buy widgets at Foobar's retail outlets.

You are in charge of a marketing drive to increase Foobar's membership. You have limited funds and reach, so you need to carefully select people from the general population that you believe will most likely become members if you target them with a promotional campaign. In this scenario you have a database of known Foobar member's (the labelled, positive class data, which we label POS) and a database of known non-members from the general public. This is your unlabelled data that contains people who may or may not become members if targeted. All we have is POS instances and UL instances -- there are no labelled NEG instances.

Depending upon your particular problem, context, and domain, you may be able to transform the OCC/PU problem into a more classical problem with standard performance metrics. In the situation I just described, there are at least three ways of creating an implicit negative class:

  1. Use the spy technique to generate "reliable" negative instances that can be placed in your test set;
  2. If historical campaign data is available you could identify those individuals who were targeted multiple times for campaign membership, but who rejected it every time -- these could be "consistently negative" instances;
  3. If your known membership pool consists of members of varying value to the business, you could divide members into good/bad, and use the bad instances as a proxy for NEG instances.

One final area you may want to explore is evaluation in collaborative filtering research. Collaborative filtering on unary responses (e.g., Facebook "like")

can be considered the matrix completion analog of the positive-unlabelled (PU) learning problem in classification and regression modelling. (Aggarwal, 2016)

Perhaps this field has developed some novel approaches to OCC evaluation that you can leverage. Without knowing the details of your particular case, however, its hard to provide a more detailed solution. Please update this thread with your eventual solution -- there is so much more research that needs to be done in this particular problem domain.

  • $\begingroup$ Thanks for such an elaborated reply. I have gone through Janssens' dissertation earlier and again revisited the mentioned chapter by you. However as per my understanding, he didn't propose any new method other than proposed by Tax in DD Toolbox, to select optimal parameter for one class classifiers. I may have overlooked if it is there. Please point out the page number if you find. $\endgroup$ – Chandan Gautam May 22 '16 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.