1
$\begingroup$

Premise:

  • Classification problem
  • Input is three text fields
  • Output classes are A, B, A&B (Note: A and B are not always exclusive though usually are, hence the 'A&B' class)
  • Sci-Kit Learn is the currently used ML library

Model:

  • Each text field is put through a HashingVectorizer (Note: ngram_range=(1,2))
  • The output is then fed to a LinearSVC (note: all experimentation with SVC and it's various kernels either never finishes or produces worse results)
  • F1-score output (rounded) is A: 0.98, B: 0.95, A&B: 0.80

Problem:

  • Some of the data lacks information on which class it is, either due to being low quality, that information being left out deliberately, or one or two of the fields being an empty string
  • Those data samples are being classified as A/B/A&B even when they are actually unclassifiable
  • Attempts to label the unclassifiable examples and add 'unclassifiable' as a fourth class have failed, in that they've resulted in low scores for all four classes as a result

Extra information:

  • Outlier detection does not detect these samples in any useful way, as far as I have been able to determine

Question(s):

  1. How do I handle the unclassifiable input data?
  2. How could I filter for it before passing it to the model for classification?
  3. How could I determine if an output classification should be discarded?
  4. If the solution involves moving from multi-class to multi-label - why? How does that solve the problem without introducing the uncertainty of effectively swapping one production model for another?
$\endgroup$
5
  • $\begingroup$ empty data are easy to identify and filter out. The rest possibly can be filtered based on the result they produce, if result for all classes is very low, discard $\endgroup$
    – Nikos M.
    Oct 21 '21 at 17:11
  • $\begingroup$ I have updated the question to clarify that there will always be at least one non-empty field. Is there any suggested way of determining how close to the hyperplane to draw the line when filtering them out? $\endgroup$ Oct 22 '21 at 8:12
  • $\begingroup$ I guess you can use same threshold as when assigning to a specific class, if not already doing so (and simply assign to largest probability, no matter its absolute value), use it. Threshold can be determined empirically $\endgroup$
    – Nikos M.
    Oct 22 '21 at 9:18
  • $\begingroup$ Do you think multi-labelling is a particularly good solution to this kind of problem, with labels A and B, and then the output of that model being used to determine if something is class A, B, A&B, or unknown? $\endgroup$ Oct 22 '21 at 9:48
  • $\begingroup$ It is an option you can try, although if data are unclassifiable, same problem re-appears in different form $\endgroup$
    – Nikos M.
    Oct 22 '21 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.