One-Class discriminatory classification with imbalanced, heterogenous Negative background?

Question

I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not.

There are about 1,150 known "positives", against a background of about 13 million protein sequences ("Unknown/poorly annotated background"), or about 100,000 reviewed, relevant proteins, annotated with a variety of properties (but very few annotated in an explicitly "negative" way).

My previous implementation looked at this as a binary classification problem: Positive set = Proteins marked as Neuropeptides. Negative set: Random sampling of 1,300 samples (total) from among the remaining proteins of a roughly similar length-wise distribution.

That worked, but I want to greatly improve the machines discriminatory abilities (Currently, it's at about 83-86% in terms of accuracy, AUC, F1, measured by CV, on multiple randomly sampled negative sets).

My thoughts were to: 1) Make this a multiclass problem, choosing 2-3 different classes of protein that will definetly be negatives, by their properties/functional class, along with (maybe) another randomly sampled set. (Priority here would be negative sets that are similar in their characteristics/features to the positive set, while still having defining characteristics) . 2) One class learning - Would be nice, but as I understand it, it's meant just for anomaly detection, and has poorer performance than discriminatory approaches.

*) I've heard of P-U learning, which sounds neat, but I'm a programming N00b, and I don't know of any existing implementations for it. (In Python/sci-kit learn).

So, does approach 1 make sense in a theoretical POV? Is there a best way to make multiple negative sets? (I could also simply use a massive [50K] pick of the "negative" proteins, but they're all very very different from each other, so I don't know how well the classifier would handle them as one big , unbalanced mix). Thanks!

Answer 1

One class learning

I wouldn't be too quick to throw out one-class classification methods (option 2) - the key is to model the positive (minority) class with the one-class model.

There has been research demonstrating cases where one-class classification out-performed other approaches like sampling for highly imbalanced data as often seen with protein classification tasks.

I couldn't find the research I recalled, but I did find some other comparisons, showing using one-class classifiers (typically modeling the minority class) achieved as good or better performance than binary classification typically with sampled "negatives" from the large set of proteins not known to be positive.

Additionally this approach also gives the advantage of much improved run-time - since you only need to train the classifier on the smaller, positive set. A couple papers:

"Prediction of protein-protein interactions using one-class classification methods and integrating diverse biological data"

"A One-Class Classification Approach for Protein Sequences and Structures"

At the very least I would try some one-class methods and compare the performance using validation with your binary/multi-class classification approaches. There are also open source implementations for many of these so it shouldn't be too costly to try them out, for example LibSVM has a one-class SVM implementation. Additionally, it might prove valuable for use in an ensemble with binary classifiers, since there may be more disagreement in their predictions.

Higher level representation embedding / clustering

Along the lines of what you were thinking with (1) and the other post suggesting PCA, approaches like clustering, sparse coding, or even topic modeling - treating each protein as a document string and different protein families as different topics - could yield a representation that might make classifying the proteins straightforward.

I.e., you could identify which group/cluster a protein belongs to or classify the cluster memberships / embedded representations.

E.g., such embedding approaches as sparse coding can yield representations that reveal which cluster a protein belongs too - so that some sets of features are only active (non-zero) for proteins in the same cluster - which can make classifying them much easier.

Additionally class labels or known cluster membership can be incorporated in the embedding process for most methods.

Ensemble

Ensembles of multiple classifiers tend to work best - especially when the classifiers are very diverse and can achieve comparable performance individually.

There are at least two ways use ensembles for this problem.

1. You can build an ensemble of binary classifiers by sampling multiple different same-size negative sets and training a classifier on each.
2. You can build an ensemble from different approaches, such as binary classifiers with different negative samples, combined with a one-class classification approach, combined with classification models trained on the embedded data.

Answer 2

The way I would attack the problem, in general, is to leverage statistical analysis like Principal Component Analysis or Ordinary Least Squares to help determine what attributes within these protein sequences are best suited to classify proteins as Neuropeptide hormone precursors.

In order to do that, you'll have to convert the protein sequences into numeric data, but I believe some work has already been done in that regard using formulas leveraged in Amino Acid PCA.

See these two links:

• Principal components analysis of protein sequence clusters
• Amino Acid Principal Component Analysis (AAPCA) and its applications in protein structural class prediction

Once that work has been done, I would attempt to classify using the entire dataset and a reinforcement learning algorithm, like Naive Bayes while slimming down the data into that which PCA has identified as important.

The reason I would try to use Bayes is that it has proven to be one of the best methods for determining spam vs. regular email, which has a similarly skewed dataset.

Having said all of that...

Slimming down the number or type of negative classifications might skew your results a few points one way or the other, but I don't think you'll see the long term effectiveness change substantially until you do the leg work of determining how to best remove the fuzziness from your training data. That will either require a field expert or statistical analysis.

I could be completely off base. I am interested in seeing some other answers, but that is my 2 cents.

Answer 3

There are three approaches you could take: APP, OAA, AAO. APP is discrimination between objects based on all possible pairs of classes. OAA is use of one-against all (remaining classes), and AAO is all at once such as use of an F-test for multiple classes simultaneously (or Hotelling's test for MVN). APP and AAO are actually multiple binary classification runs but with more than two of your original classes. Each of these approaches yields different results for various classifiers employed.

Random sampling is a good technique. You might also try to cluster all of the objects into centers using k-means, and then use the centers as new objects. Either way, linear and non-linear dimension reduction methods might help get away from the large sample size.