0
$\begingroup$

was hoping to ask how to approach setting up a train test split for a dataset that is provided in two separate .csv files: one as the "train" dataset, and the other as the "test" dataset.

I've been taught to utilize sklearn's train_test_split for usually one dataset that goes ahead and splits it into the respective X train/test, y train/test, but I can't seem to find any documentation on the approach if the datasets are fed in as two separate data frames.

Would the best approach be the merge the two back together and then apply train_test_split?

Thanks for the help in advance!

$\endgroup$

2 Answers 2

0
$\begingroup$

In cases where the split is already defined (e.g. by two files or by an extra column), you do not need to apply train_test_split, just use the given split.

For you this would look something like that (assuming you have a function load_dataset:

X_train, y_train = load_dataset("train.csv")
X_test, y_test = load_dataset("test.csv")

Nevertheless, I might be a good idea to perform cross-validation on the training dataset for hyper-parameter optimization.

$\endgroup$
1
  • $\begingroup$ Perfect this makes sense, thanks so much for the help @Broele !! Much appreciated $\endgroup$
    – acboy
    Apr 7, 2023 at 14:49
0
$\begingroup$

I'm unclear as to why you would want to merge the two datasets back together and then resplit them. This certainly can be done, for example if they both have the same column names:

new_dataset  = pd.concat([train_set, test_set], axis=0)
from sklearn.model_selection import train_test_split
train_set, test_set = train_test_split(new_dataset  , test_size=0.2, random_state=42)

Do you know that there's something wrong with the way the existing data is presplit before you merge them together? I.e. test set too small, not shuffled, not stratified sampled, or something else? If there's no reason to join them back together, leaving them separate and using as is might be the easiest approach.

$\endgroup$
1
  • $\begingroup$ Thanks so much for the response @brewmaster321! I don't believe there's anything wrong with the existing data, but I was just trying to determine what the best way to go about assigning them to the X train/X test and y train/y test would be as I've only learned the one way of train test splitting a datafrom sklearn before preprocessing for a model. I'm still new to the field and wasn't sure how to handle these variables when a data set is already split for you as I believe this was done in relation to a research study so the authors I'm assuming wanted to show what their data was. $\endgroup$
    – acboy
    Apr 7, 2023 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.