1
$\begingroup$

I have a maybe naive question about the appropriateness of using binary classifications. This is a hypothetical example, so forgive me if it is too coarse.

Let's say I want to train a support vector machine to classify the letter "a" from the letters "b" and "c". The problem is that I don't have letter "a" to train my SVM model, but I do have a lot of letters "A" which basically mean the same. It is a model like this valid?

My real problem is within the realm of image processing. I have some pixels that I want to classify using texture but I don't have a "class" for those pixels. I do have ground truth for something very similar (that I know the class) and I have been using the same kind of texture (but from this similar pixels) to train a SVM and do the classification. This SVM works reasonably well (assessed with the visual inspection of the images) with some accuracies up to 90%. The problem I think is that this accuracy is tested with the training of "A" and not "a"

Does this make any sense?

Is something like this valid? or there is anything else that I can do to make this a valid classification? How do you test accuracy in something like this?

$\endgroup$

1 Answer 1

0
$\begingroup$

This is the issue of evaluation, and the answer is always the same: if the goal of the system is to be be applied to detect "a", then it must be evaluated on a representative dataset containing "a" values. Practically, this usually implies that a subset of the real target data must be manually annotated (preferably by several annotators), so that it can be used as test set.

There's no problem training the model with any dataset, for example with "A" values, or build a rule-based model manually, or use a random method... The way the model is trained doesn't matter, as long as it works as intended. To test this, the model must be evaluated on the real target task. If the model is evaluated on data with "A", at best you prove that the system detects "A", but you don't prove anything about "a". The alternative would be to independently prove that detecting "A" is equivalent to detecting "a", but proving this would require an annotated dataset with "a" values anyway, so it's not easier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.