1
$\begingroup$

I am applying a ML model (LGBM binary classifier) to data and would now like to identify the part of data where I have a low ratio of false-negatives (false-postives are not such a problem) and as much as possible true-negatives.

Background

The data I am classifying is from a system, that includes many complex rules that decide whether or not for some cases the company pays goodwill/warranty. In many of the cases this rule based system already finalizes the decision and initiates the payments. However there are cases, where this rule based system isn't able to finalize a decision but instead forwards the case along with a proposed decision to a manual decision process. It seems that this often also happens, because the rules (which are highly configurable) are not sharp enough or because the price-depending parts are not adjusted to inflation etc.

Now in this configuration it turns out, that the manual decision process very often just accepts the proposed decision. I am working on training a ML model (the LGBM classifier) to identify the cases, where the manual decision process leads to just accepting the proposed decision without changeing it.

Motivation

I am only interested in identifying cases where the proposed decision made by the rule system can just be taken over, because I'd like to reduce the amount of decisions that have to be checked manually. My model currently gets a MAE of around 0.8 at the moment.

I'd now like to find a segment of the data, where the ratio of false negatives (the decisions where the model states, they need not change but in fact the descider changed them) is smallest (or below a certain threshold --> say 0.02).

Question

Can anybody suggest, how I could find such a region with a low ratio of false-negatives (false negatives in region by number of samples in the region)? I thought about a simple decision tree, but would like to know if there are better options.

E.g. I suspect, that in case a leaf contains a high percentage of say 95% class 1 samples and 5% class 0 and some condition would allow to split it in two regions with 98% class 1 and 92% class 1 it wouldn't perform that split, since both would be classified as class 1 anyways, or am I wrong here?

Improve this question
$\endgroup$
5
  • $\begingroup$ How big is your dataset that you are using? $\endgroup$
    – khwaja wisal
    Commented Oct 3, 2019 at 21:35
  • $\begingroup$ It's about 500.000 rows and 1230 columns. $\endgroup$
    – jottbe
    Commented Oct 4, 2019 at 3:55
  • $\begingroup$ @khwajawisal thank you for your suggestion. I already tried neural nets but they didn't perform better. Maybe something is missing in my data. $\endgroup$
    – jottbe
    Commented Oct 4, 2019 at 19:49
  • $\begingroup$ I think your dataset is big enough to try Neural networks and get significant results, you can visualize your layers to see how things are being classified and there are a whole lot of tricks which you can use to improve your performance and also have you tried RFECV because 1230 attributes they are a lot try RFECV with different Classifiers to get that optimal subset of features because I personally feel there is a lot of scope of feature selection here because too many redundant features or correlated features make modeling hard and interpreting the results even harder. $\endgroup$
    – khwaja wisal
    Commented Oct 4, 2019 at 19:54
  • $\begingroup$ why don't you try ensemble methods for feature selection you can use classifiers various functions such as feature_ranking and feature_importances try to use them concatenate them average them or use weighted voting there are many tricks and you have to find what the best technique that can explain your data and the results you aquire using it. $\endgroup$
    – khwaja wisal
    Commented Oct 4, 2019 at 19:59

2 Answers 2

Reset to default
1
+50
$\begingroup$

A way to explore your dataset would be to use different techniques and see how they preform. The first approach I would consider is a tree based learner of modest complexity (or at least not boosted), like C4.5. It's possible (or perhaps likely) that won't perform as well as some more powerful approaches, but it has the added benefit of being relatively readable.

If you are familiar with a decision tree visualisation, you'll know that you can identify (leaf)nodes with high(er) impurity, and a large(r) population quite quickly. You can find an example of such a visualisation here. Those nodes can help you understand your domain, and possible how to improve performance (For instance by boosting, something LGBM already does for you).

If you want to optimise the LGBM straight away, another thing that could help here is to cluster your dataset (kMeans could be a good starting point) and then calculate performance metrics per cluster. Possibly you could also consider clustering the scored items, including the classification, or even the evaluation (TP, FP, TN, FN). If used right that could help identify clusters of false negatives for instance, and the commonality they share. Disclaimer: I have never done this myself.

PS: Lot's of times I find it helpful to use KNN (over a subsample of the feature space) and Naive Bayes. They can quickly tell you something about the global correlations. Other golden oldies are feature importance rankers and a correlation plot.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi, thank you very much for your answer. I'll check your proposals. It sounds quite promising. I also already thought about KNN, but does it also work with predefined labels? I only know it as unsupervised learning method. Or would you only use it to cluster the data and then train a separate LGBM model on each of the clusters? $\endgroup$
    – jottbe
    Commented Oct 1, 2019 at 8:24
  • $\begingroup$ K-Nearest-Neighbor is a supervised method, K-Means is an unsupervised method. I suggested you use kNN to explore how the relations work, ut you could use kMeans over the dataset included the classifications and evaluations (not to be confused with supervised learning) to find clusters of faults and patterns. $\endgroup$
    – S van Balen
    Commented Oct 1, 2019 at 9:12
  • $\begingroup$ Ah you're right I confused KNN and K-Means. I already use feature selection btw and am aware that I get a purity measure from a decision tree, but want to avoid overfitting by excluding a larger set of very small regions where my model doesn't work. But I'll try according to your proposal if I can approach this with a flat C4.5-tree or naiive bayes. $\endgroup$
    – jottbe
    Commented Oct 1, 2019 at 10:14
  • $\begingroup$ When I wrote this question, I hoped for getting a specialized method which identifies such regions of high purity. I was skeptic if decision trees are the correct means to dot that. In the end I used a regular decision tree and found out that there are probably no such reagions of "high purity" where I can restrict my ML model on. It seems I have to search for additional infos so my model can decide better. $\endgroup$
    – jottbe
    Commented Oct 7, 2019 at 16:20
1
$\begingroup$

Maybe I'm not understanding 100% what exactly you mean by regions, but assuming your model outputs well-calibrated probabilities, you can directly read these "regions" off of the predictions. Among all instances with probability greater than $98\%$ there will be fewer than $2\%$ of negatives (basically by definition).

An LGBM classifier should be able to produce well calibrated probability estimates using (by default) a cross-entropy log-loss in the binary setting. You can check visually that they are indeed well-calibrated. And in the case that they are not, you can explicitly calibrate them by fitting a "meta" model over the probability predictions. The scikit-learn documentation has a nice description of probability calibration.

If you need to figure out intuitively what the cases are that can be taken over, you can just bucket the cases based on their predicted probability and do some exploratory analysis.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi oW_ thank you for your explanation. I'll check out the description. I want to find out well-definable conditions I can apply to my input data, so I am sure my model has the desired accuracy over this region. I think if I phrase it different, then what I am searching is a subspace of the vector space in which my sample lives in. this subspace of course does not have to be a vector space itself. Just something which can be identified. Either by conditions (e.g. if my predictor works well for costs < 500 EUR but makes more errors above, that probably would be a good split). $\endgroup$
    – jottbe
    Commented Oct 3, 2019 at 7:05
  • $\begingroup$ I know, that sounds very much like a decision tree, but I think a plain vanilla decision tree is constructed to produce the best prediction over the whole data set and what I need is something that would optimize for two things at the same time. So ideally it would look for the largest subspace (defined by conditions), where the accuracy of the model is at least say 98%. Or even better: for the k largest subspaces, on which the accuracy of 98% can be achieved. $\endgroup$
    – jottbe
    Commented Oct 3, 2019 at 7:05
  • $\begingroup$ Thank you very much for your info. Unfortunately I have to select one answer only and the one of S van Balen contains a bit more suggestions. In the end I used a regular decision tree and found out that there are probably no such reagions of "high purity" where I can restrict my ML model on. It seems I have to search for additional infos so my model can decide better. $\endgroup$
    – jottbe
    Commented Oct 7, 2019 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.