1
$\begingroup$

Generally if one dataset is given we use

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

y_pred = lr.predict(X_test)

print(confusion_matrix(y_test,y_pred))

print(accuracy_score(y_test,y_pred))

print(classification_report(y_test,y_pred))

if we are doing validation on the training dataset.

X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.3, random_state=0)

But where and how to use the test dataset in the code?

$\endgroup$
1
  • $\begingroup$ Training data has to be single dataset. Make sure that test and validation dataset have same shape $\endgroup$
    – amol goel
    Aug 21, 2022 at 7:36

2 Answers 2

2
$\begingroup$

Just to make sure that everyone is on the same page a short review what train, validation and test are good for:

  • The Train Set is used to train a concrete model with concrete hyper-parameters
  • The Validation Set is to measure and compare the performance of different models, especially, models that result from different hyper-parameters
  • The Test Set is set aside and only used when you are done with training models and optimizing hyper-parameters

So in case you have all data in one dataset, you might want to do something like this:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.3, random_state=0)

to get all three sets. Then you would do all the training / optimizing on X_train,y_train and X_valid,y_valid and just come back to X_test,y_test at the very end.

If X_test,y_test are already given in a separate data-set, then you can skip the first splitting part, only split into train and validation and plainly load the test-set.

I hope this answers your question?

$\endgroup$
1
$\begingroup$

We are commonly using 3 main types of Data Sets when we train the Neural Network Model. Train, Dev and Test Set.

  • Train Set is used to train the model.
  • Dev Set (Hold-out cross validation) is used to evaluate trained model and tuning the Hyperparameters during the development process.
  • Test Set is just used measure final performance of the final system as production level.

FYI, difference between Dev and Test sets is the Dev Sets is used on the training the model, instead where the Test Sets only used to measure model's performance before deployment on real project.

Train / Dev / Test Sets Distributions

  • Train, Dev and Test sets should from same distribution and must be taken randomly from all the data.
  • Choose Dev and Test set to reflect data you except to get in the future and consider important to do well.

Size of Train, Dev and Test sets

  • Setup the size of Test sets to give high confidence in the overall performance of the system.
  • Dev sets has to be big enough to evaluate different ideas.
  • If your dataset is small one, you can separate whole data into 60 / 20 / 20 or 70 / 15 / 15 would be good.
  • If your dataset is enough to big such as millions or hundred of millions, then you can separate whole data into 98 / 1 / 1 or 99 / 0.5 / 0.5.

These theory is comes from one of the best Deep Learning course - DLS(Deep Learning Specialization) on Coursera.

Hope this give you intuition how to use Train / Dev / Test sets to train your system as well. Best Regards.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.