How does the test data gets collected?

Question

I'm learning the linear regression now. I used R to build linear model upon a set of train model and try to predict() data based on the test data.

My question: I understand how train data gets collected. But what's with the test data? How did it get collected? Is the test data built, or is it collected, or is it predicted or what?

PS: I'm learning data science by method of self-learning, so I lack the structural in my knowledge. I might know something on one place while lack the knowledge at another place. Please forgive and guide. Thanks. :)

Answer 1

In general, you randomly splits the available data into following 3 sets. Train, validation and test. [Different ratios could be used depending on total amount of data at hand and the difficulty of the problem.] You can start with a simple 80%/10%/10% split. You use the first two sets to build your model. So you would use the 80% split of your data to lets say build a logistic regression model. Now you use the 10% validation set to see how good your model is. You can iterate on this process until you are satisfied with the validation data set performance of the model.

Now, you use the last set (the "test" split) to see how your model would generalize on an "unseen" data set. [Here unseen means, your model never had access to "see" this data while in the learning phase]. You never use use this 3rd set to glean knowledge about how your model could be improved. Think of this set as a set your customer has, and you don't have access to, and to whom you are going to deliver your model.

Once you get a hang of this concept also learn more about how you can do cross-validation if you have limited data and to improve confidence on your model.