I read about T-Test and how we can use it to compare between 2 models (https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa)

There are some issues I'm not sure I understand correctly:

  1. I saw that we usually choose P (p-value) to be 0.05 or 0.01

    1.1 What does it mean ?

    1.2 Does it mean that if the calculated t is in the range of the T-Table we accept the null hypothesis ,i.e 95% (or 99%) of the tests, there is no difference between the 2 models ?

  2. According to https://medium.com/analytics-vidhya/using-the-corrected-paired-students-t-test-for-comparing-the-performance-of-machine-learning-dc6529eaa97f

    we can't use K-FOLD and T-TEST, because the samples not independent (Train data used in one fold and in the other it is the test), am I right ?

    So to use T-TEST we must generate at least 30 different datasets for test (with no dependent) ?

  3. Degree of freedom

    3.1 If I understand correctly, more test we run (bigger N -> bigger degree of freedom) -> the more accurate the T-Test results ? am I right ?

    3.2 Do we need to run at least 30 tests to use T-TEST ?

  • $\begingroup$ 3.1 more N does not mean "the more accurate" the T-Test results ? $\endgroup$ Sep 15, 2023 at 23:54

1 Answer 1


(p-value) to be 0.05 or 0.01 means probability of rejecting a null-hypothesis on account of sampling fluctuations. Generally, we compute p-value from the sample data and compare it with the statistical table value at specified level say p = 0.05. Many a authors use the term - alpha - to reflect probability of rejecting null-hypothesis on account of sampling fluctuations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.