Reduce false positives having imbalanced data

Question

I'm using a DNN-48 having the following scenario:

• Features: 8 (48 at the end because I generate conditional sequences of 6 elements each)
• Classes: Y=0 (90%), Y=1 (10%)

Precision and recall are good when Y=0. If I adjust weights properly the recall is good on Y=1 too. The real problem is precision when Y=1.

Using SMOTE results were even worse. Best alternative at the moment was applying a custom stratification based on picking 8 of Y=0 for each Y=1 found.

I've tried moving the threshold to 0.8-0.9 and it slightly works but it sacrifices some recall. I've also tried changing the NN topology adding more layers, units and using regularization like L1/L2 and dropouts. For the time being I've seen best results using simple a topology like:

model = Sequential([
    Dense(64, activation='relu', input_shape=(X.shape[1],)),
    Dense(32, activation='relu'),
    Dropout(0.2),
    Dense(16, activation='relu'),
    Dropout(0.2),
    Dense(8, activation='relu'),
    Dense(4, activation='relu'),
    Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy', f1_score])

I'd like to keep the good recall without having too much false positives. Any ideas? thank you!

Answer 1

First of all, there will always be a trade-off between precision_score and recall_score. So you must choose a satisfying recall_score that you can live with.

There are few techniques that you might try:

Use F-beta Score: F-beta score allows you to give more importance to either precision_score or recall_score depends on beta value.

Basic rule for that:

• If beta is less than 1, this gives more importance to precision (fewer false positives) over recall.
• If beta is greater than 1, this gives more importance to recall (fewer false negatives) over precision.
• When beta equals 1, it's equivalent to the F1 score - giving equal weight to both precision and recall.

You can choose whichever value of beta that you want based on your results.

Ensemble Methods: Another approach is using ensemble methods designed for imbalance problems such as Balanced Random Forests and Easy Ensemble classifier which balance classes internally.

To create Balanced Random Forest, you need to install imbalanced-learn.

from imblearn.ensemble import BalancedRandomForestClassifier

brf = BalancedRandomForestClassifier(n_estimators=100)
brf.fit(X_train, y_train)

predictions = brf.predict(X_test)

Answer 2

Is this tabular data that you are exploring? If so, how many different models have you compared the results with?

What is the variance of your precision and recall for both class given different splits of your data?

All of that might help to get a better sense - augmenting the less represented class alone might not help in some cases but more details would be great to provide you a better answer.

Answer 3

One issue with SMOTE is overgeneralization: in a nutshell, it creates new points that are in the line joining two existing points of the minority class. This helps in some situations, but it might not generate enough variability for your classifier to learn properly. Have you tried using synthetic data for oversampling? It helps maintain your data characteristics better than SMOTE.