1
$\begingroup$

I've built a binary classification model based on Keras and I am getting about 70% accuracy, and about 72% precision and 88% recall, making up to 79% F1-Score. I've tried different data models (deriving new features (columns) from the original ones), but the same model (same hyperparameters, same number of hidden layers (1), and same number of nodes in each layer).

For the different models created, after evaluating, the values of accuracy, precision, recall and F1-Score are almost the same as above. However, the Recall was always (for all models) high for all of the models tested, ranging from 85% to 100%.

What does that say about my model? Is it good enough? Why is the Recall for all models so high, but precision around 61% and 73%?

PS: When I mean "all models", as I said above, it is the same keras model, with same layers and nr of nodes and hyperparameters, just trying different dataframe as input. The first one is the original one, and the others have for example 1 feature less, one that I think is not relevant. Or another model, with input data same as original one, but with adding a derived feature, for example adding a feature "is_weekend", denoted with 1 or 0, but this feature is derived from the date feature already available in the original data. And for me, it's another model, as the input data is not the same.

Here are the evaluation results for my first model, the basic dataframe, with original features, no derived columns in it:

basic_df
accuracy:  0.7068636796381271
f1-score:  0.7915721938539619
precison:  0.6974898160810807
recall:  0.9234265146423681

My dataset in not strongly imbalanced. I have 6992 data points, with 4303 class 1 (61.54%) and 2689 class 0 (38.46%).

Loss over epochs accuracy

$\endgroup$
  • 1
    $\begingroup$ Could you plot the ROC curve using the probabilities? That could give you better insights. $\endgroup$ – Danny Feb 11 '19 at 15:57
  • 1
    $\begingroup$ I agree with Danny - an ROC curve would be helpful. Having a high recall isn't necessarily bad - it just implies you don't have many false negatives (a good thing). It's similar to precision, higher typically is better. It's just a matter of what you care about more: false positives (precision) or false negatives (recall). There are other metrics which give a more holistic view, F1 is one, but certainly not the best. $\endgroup$ – Alex L Feb 12 '19 at 3:51
  • $\begingroup$ It's generally happened when a model is biased towards one class. In such a case you will get high Recall and low precision towards majority class. Good to review the training dataset as well and if data is not proportional, I think getting more data will help you in preparing better model. $\endgroup$ – vipin bansal May 28 '19 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.