The meaning of P and degree of freedom in T-Test

Question

I read about T-Test and how we can use it to compare between 2 models (https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa)

There are some issues I'm not sure I understand correctly:

1. I saw that we usually choose P (p-value) to be 0.05 or 0.01

   1.1 What does it mean ?

   1.2 Does it mean that if the calculated t is in the range of the T-Table we accept the null hypothesis ,i.e 95% (or 99%) of the tests, there is no difference between the 2 models ?

2. According to https://medium.com/analytics-vidhya/using-the-corrected-paired-students-t-test-for-comparing-the-performance-of-machine-learning-dc6529eaa97f

   we can't use K-FOLD and T-TEST, because the samples not independent (Train data used in one fold and in the other it is the test), am I right ?

   So to use T-TEST we must generate at least 30 different datasets for test (with no dependent) ?

3. Degree of freedom

   3.1 If I understand correctly, more test we run (bigger N -> bigger degree of freedom) -> the more accurate the T-Test results ? am I right ?

   3.2 Do we need to run at least 30 tests to use T-TEST ?

Answer 1

(p-value) to be 0.05 or 0.01 means probability of rejecting a null-hypothesis on account of sampling fluctuations. Generally, we compute p-value from the sample data and compare it with the statistical table value at specified level say p = 0.05. Many a authors use the term - alpha - to reflect probability of rejecting null-hypothesis on account of sampling fluctuations.