Binary problems have the most amount of metrics to measure its performance.
They can be classified as accuracy metrics, probabilistic metrics and metrics depending on true/false positives/negatives. You are not expected to implement them from scratch, the last time I implemented an AUC (area under the curve) was in college so you will find these metrics already implemented in most machine learning libraries in all languages. This is a comprehensive list taken from the Keras library, check the links for more information about the metrics or the library.
About the matter of the outputs you can have a true/false 1/0 values or the probabilities of the observation to belong to a class which is a better way to measure the algorithm's performance. Do you need a threshold to evaluate results properly, it all depends on the data you have. For some problems high accuracy like above 99.999% is critical like clinical diagnoses but for others accuracy like 55% can be seen as great like the case of stock predictions. So it depends on your data.
Accuracy metrics:
These metrics, simple in design, are often raw percentages of the correctly labeled classes for one class or many classes.
Raw Accuracy
Binary Accuracy
Categorical Accuracy - Straightforward accuracy for all classes.
Top K Categorical Accuracy - Accuracy for the top K categories
Sparse Top K Categorical Accuracy
True False Positive Negative metrics:
More expressive metrics useful to determine the impact the algorithm has not only in its accuracy but also in the wrongly classified observations.
AUC
Precision
Recall
True Positives
True Negatives
False Positives
False Negatives
Precision At Recall
Sensitivity At Specificity
Specificity At Sensitivity
Probabilistic metrics:
These as often used as the function loss for machine learning algorithms, however, they can also be understood as a benchmark metric specially for a validation set.
Binary Crossentropy
Categorical Crossentropy
Sparse Categorical Crossentropy
KLDivergence
