I have a table of 2 features (numbers) on which I define a simple binary classification "model" (i.e. a simple logical expression) which needs 2 parameters thresholds. The model tries to predict "events", i.e. whether an event has happened in the past or not. I also have s set of labels (observations of past events) which are known to be incomplete, meaning that if the label is 1 it has been observed, if its 0 it means we just don't know anything (e.g. event happened, but was not recorded)
|feature A||feature B||label|
My model would to something like
if (feature_A < tresh_A or feature_B < thesh_B) then 1 else 0
Due to the "sparseness" of the test data, I can only calculate recall, but not precision (as I don't know the true negatives, only true positives). Well I could calculate precision, but false-positives would be way too high because many of them would actually be true positives, but just missing in the obersvations....
Any advice on how to optimize the parameters in such a case? I could easily do a grid-search as it's computationally cheap, but I don't know what to optimize for. Intuitively, I would like to maximise recall and at the same time minimise overall positives positive prediction.