1
$\begingroup$

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model?

I have a binary model that I have developed with a NN in Keras. I know the size of my (equally balanced) training set and it has a proportion of 0.5 (duh!). I know that with my business use case, false-positives are financially expensive so I'm focusing on Precision as my metric. So, in validation, can't I take that Precision metric as a proportion of my validation set (which I also know the size of) and then get the z-test score calculation? That should give me a threshold for validation Precision at which my model is doing more than just flipping a coin.

Can someone confirm my line of thinking or am I way off base here?

$\endgroup$
3
  • 1
    $\begingroup$ Do you mean a z-test? // You might be interested in proper scoring rules and the probability outputs of your neural network. Frank Harrell's blog discusses this topic: 1 2. $\endgroup$
    – Dave
    Jun 21, 2021 at 15:23
  • $\begingroup$ @Dave Fair point. I have edited my question $\endgroup$ Jun 21, 2021 at 15:25
  • 1
    $\begingroup$ Why not just take every observation as a negative? Then you have zero false positives? (I get why this will get you fired, but consider why this is not a viable business solution.) $\endgroup$
    – Dave
    Jun 21, 2021 at 15:35

1 Answer 1

0
$\begingroup$

From your description, you can not use Z-test because the z-test requires knowing the population variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.