I am applying a ML model (LGBM binary classifier) to data and would now like to identify the part of data where I have a low ratio of false-negatives (false-postives are not such a problem) and as much as possible true-negatives.
The data I am classifying is from a system, that includes many complex rules that decide whether or not for some cases the company pays goodwill/warranty. In many of the cases this rule based system already finalizes the decision and initiates the payments. However there are cases, where this rule based system isn't able to finalize a decision but instead forwards the case along with a proposed decision to a manual decision process. It seems that this often also happens, because the rules (which are highly configurable) are not sharp enough or because the price-depending parts are not adjusted to inflation etc.
Now in this configuration it turns out, that the manual decision process very often just accepts the proposed decision. I am working on training a ML model (the LGBM classifier) to identify the cases, where the manual decision process leads to just accepting the proposed decision without changeing it.
I am only interested in identifying cases where the proposed decision made by the rule system can just be taken over, because I'd like to reduce the amount of decisions that have to be checked manually. My model currently gets a MAE of around 0.8 at the moment.
I'd now like to find a segment of the data, where the ratio of false negatives (the decisions where the model states, they need not change but in fact the descider changed them) is smallest (or below a certain threshold --> say 0.02).
Can anybody suggest, how I could find such a region with a low ratio of false-negatives (false negatives in region by number of samples in the region)? I thought about a simple decision tree, but would like to know if there are better options.
E.g. I suspect, that in case a leaf contains a high percentage of say 95% class 1 samples and 5% class 0 and some condition would allow to split it in two regions with 98% class 1 and 92% class 1 it wouldn't perform that split, since both would be classified as class 1 anyways, or am I wrong here?