# Purpose of test data in binary classification

I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as Accuracy, F1Score, and etc.

UserId,UserName,AppId,Label
1,Lin,1,1
1,Lin,2,1
1,Lin,3,1
1,Lin,4,1
1,Lin,5,1
1,Lin,6,0
1,Lin,7,0
1,Lin,8,0
1,Lin,9,0
1,Lin,10,0
2,Tim,1,1
2,Tim,2,1
2,Tim,3,1
2,Tim,4,1
2,Tim,5,1
2,Tim,6,0
2,Tim,7,0
2,Tim,8,0
2,Tim,9,0
2,Tim,10,0
3,Kirstin,1,1
3,Kirstin,2,1
3,Kirstin,3,1
3,Kirstin,4,1
3,Kirstin,5,1
3,Kirstin,6,0
3,Kirstin,7,0
3,Kirstin,8,0
3,Kirstin,9,0
3,Kirstin,10,0
4,Michael,1,1
4,Michael,2,1
4,Michael,3,1
4,Michael,4,1
4,Michael,5,1
4,Michael,6,0
4,Michael,7,0
4,Michael,8,0
4,Michael,9,0
4,Michael,10,1
5,Timmy,1,0
5,Timmy,2,0
5,Timmy,3,0
5,Timmy,4,1
5,Timmy,5,1
5,Timmy,6,0
5,Timmy,7,1
5,Timmy,8,1
5,Timmy,9,1
5,Timmy,10,0
6,Lick,1,1
6,Lick,2,0
6,Lick,3,0
6,Lick,4,0
6,Lick,5,1
6,Lick,6,1
6,Lick,7,0
6,Lick,8,0
6,Lick,9,1
6,Lick,10,0
7,Shawn,1,1
7,Shawn,2,0
7,Shawn,3,1
7,Shawn,4,0
7,Shawn,5,1
7,Shawn,6,0
7,Shawn,7,0
7,Shawn,8,0
7,Shawn,9,1
7,Shawn,10,1


To test the trained model, I used this test set.

UserId,UserName,AppId,Label
8,Lianne,1,1
8,Lianne,2,0
8,Lianne,3,1
8,Lianne,4,0
8,Lianne,5,0
8,Lianne,6,0
8,Lianne,7,0
8,Lianne,8,0
8,Lianne,9,1
8,Lianne,10,0


Following is the prediction results.

8,Lianne,1,1  => Predicted true
8,Lianne,2,0  => Predicted false
8,Lianne,3,1  => Predicted true
8,Lianne,4,0  => Predicted true
8,Lianne,5,0  => Predicted true
8,Lianne,6,0  => Predicted false
8,Lianne,7,0  => Predicted false
8,Lianne,8,0  => Predicted false
8,Lianne,9,1  => Predicted false
8,Lianne,10,0 => Predicted false

// Sum this up...
TP = 2
TN = 5
FP = 2
FN = 1

Accuracy = 2 + 5 / 2 + 5 + 2 + 1 => 7 / 10 => 0.7
Recall = 2 / 2 + 1 => 0.66
Precision = 2 / 2 + 2 => 0.5
F1Score = 0.5 * 0.66 / 0.5 + 0.66 * 2 =  0.33 / 1.16 * 2 = 0.5689


The calculation was proven alright and as I expected, the model is right on almost all cases except the app id 2. While I'm doing this, this idea comes into my mind.

" What test data is for? "

" Is it for checking out the desired model metrics like above? "

" Why bother with some proportion of data to perform a test? when you just need a certain, small group of dataset to evalute the model? "

" Even if I have a thousands of million rows of training dataset, wouldn't it be enough to have a complete small, unique set of dataset that covers the whole variations? do I still need some proportion of testdata from the training set? "

Well, if I already know that the app id range is from 1 to 10, isn't it fairly enough to just prepare only 10 rows of test set as long as it has app id 1 to 10? even when the training data set is gigantic?

Why do I need to care about N% of test data set from the training data set?