Testing a Binary Classifier

I have been training a binary multilayer perceptron on a database made out of roughly 3600 0 values, and 4 1 values. Afterwards, I'm testing the MLP on a test set made out of 7 0 values and 7 1 values. The little amount of 1's in my database is due to the fact that the data collection of this class is rather hard. My MLP is yielding good results, however, my question is if I can interpret these results as good, or if the distribution in classes might have an effect on this. I do not want to overhype my results, and that is why I'm checking over here!

• I do not believe you have enough observations of the $1$ to produce a useful model. Even $3600$ observations of $0$ is pushing it for deep learning.
– Dave
Jun 15 at 14:51
• Also, I have concerns about you simply not collecting many observations of $1$. Is that because $1$ is rare or because $1$ is inconvenient? Rare classes are not a problem when you use proper scoring rules like log loss or Brier score (not discontinuous, improper scoring rules like accuracy, sensitivity, specificity, or $F_1$ score), but by not collecting inconvenient data, you would be giving the model the wrong prior probability of membership in $1$, which will influence the posterior probability predicted by your model.
– Dave
Jun 15 at 15:00
• Why is the test set made of the same proportion of 0 and 1? The test set should follow the distribution expected in some real sample of data. Also 14 instances is a very small test set, but I guess this is related to the first point. Jun 15 at 17:26
• It is for a project and the deadline is soon, and there is little to no time left to collect more data instances of class 1. However, why is 14 instances small, are there constraints to the size of a test set and where is this based upon? Jun 16 at 12:50