I would suggest you to:
- Balance your dataset (since you are not doing an anomaly detection task, that should be fine).
- Measure Precision/Recall/F1 Score and argue for them depending on your problem.
- Perform K-Fold Cross Validation to compare models
Balancing your dataset
You said you have a
33:67 ratio. Why keep it like that an not just make it
50:50 (e.g. by removing samples from the larger class)? In general this help your models be less biased to the larger class.
Special care is needed when doing anomaly detection, where ratios are usually around
1:99. But since you are not doing that, simple dataset balancing is advisable.
Precision / Recall / F1
F1 score is the default score to go to for classification. It should be ok to use across models as long as you keep your
test set the same for all models. An even better thing to do to compare models, is to perform
K-Fold Cross Validation on each model. (Sklean docs).
However, you could look at Precision or Recall instead of the F1 Score, but this will depend on the problem you are trying to solve.
Precision = when I detect an instance of a class, I am sure it is the right class.
Recall = I detect all instances of a class, even if I detect instances that don't belong.
For example: we are performing a binary classification of whether a patient has a deadly disease or not. Health Insurance: "let's optimize the precision, because we only want to pay treatment for patients that really have the disease". Doctor: "let's optimize for recall, because it is better to treat more people even if the don't have the disease to ensure that those that do have it are likely being taken care of".
If you think that precision are recall are equally important then that is the definition of the F1 score.