2
$\begingroup$

I have a DataFrame representing an annotated dataset with 300 labels.

The DataFrame looks like follow (the first row is just to explain the columns):

binary Vector column for labeling, feature column
[0, 1, 1, 0, 0 , 0.... ], featureVec
[0, 0, 1, 0, 1 , 0.... ], featureVec
......

The labelling column, represent the presence or the absence of each label as an annotation of a feature vector (which we can see in the second column).

Please correct me if I am wrong: I can't train one multi-class classifier because the labels are not exclusives, so I would like to train one binary classifier per label.

As I have 300 labels, then I need to train and optimise 300 classifier (say logistic regression classifier for example).

What is the best way and best practices to train my classifiers with N-cross validation for parameter optimisation? any example code or reference is highly appreciated.

Once the classifiers are optimised, what is the best way and best practices to save the best models and use them to classify the new data?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ This is not a good approach and here is there reason why. $\endgroup$
    – eliasah
    Commented Nov 4, 2015 at 22:05
  • $\begingroup$ Thanks Elisha, even though I don't agree with some points in the link you provided. I would appreciate if you can elaborate more your answer and provide a good approach. Thanks $\endgroup$
    – Rami
    Commented Nov 4, 2015 at 22:32
  • $\begingroup$ I agree that that explanation is not entirely relevant (particularly if you scale your input for the whole dataset at once and use appropriate sampling/weighting schemes). But it might still be hard to train that many classifiers. You might consider a multiple regression with your output as 0s and 1s for each label. You can then apply a threshold to your prediction to get a binary vector of labels. This also has limitations, but you get to use all your data instead of having to oversample for each individual classifier $\endgroup$
    – jamesmf
    Commented Nov 5, 2015 at 4:06
  • $\begingroup$ thanks @jamesmf, this what I am trying to do now, and actually a binary output is not a most in my case, the order of the predicted categories is what I will be looking at. I am just wondering if applying a random forest is more relevant for that scale of problems? $\endgroup$
    – Rami
    Commented Nov 5, 2015 at 8:28
  • 1
    $\begingroup$ If it's just an order, then regression is even more applicable. If you wanted to train 300 models, you could use random forest. If you wanted one 300 dimensional output, you might try a regression (or a neural net can do this). The applicability of a random forest (and decision trees in general) probably depends on the nature of your feature space. $\endgroup$
    – jamesmf
    Commented Nov 5, 2015 at 16:16

1 Answer 1

Reset to default
1
$\begingroup$

This is a general ML answer, not about Spark 1.5.1, but perhaps it can help you find a way to the solution.

Your problem is multi-label as well as multi-class (e.g. http://scikit-learn.org/stable/modules/multiclass.html).

Some algorithms have multi-label versions, and will deal with the problem internally so you don't have to. If you don't have a multi-label version of your algorithm, you can either change to an algorithm that does, or roll your own multi-label solution around your algorithm, for example using binary classifiers for each label. (This typically leads to worse performance as there are many complications and possible optimisations to take into account).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.