# Split unprocessed dataset into train and test sets

I have two files namely - log.csv and label.csv. My aim is to to split these files into training and testing datasets before preprocessing the log_train file further.

The log.csv file has a no. of columns including id. For one value of id, the file has 'z' no. of rows (this 'z' is different for different value of id) . Whereas, in the label.csv file there is just one row for each value of id. I wish to split the files into - log_train.csv, log_test.csv, label_train.csv and label_test.csv obviously such that all rows corresponding to one value of id goes either to train or test file with corresponding values in label_train or label_test file.

I have already tried doing: X_train,X_test,y_train,y_test=train_test_split(df,df_label,test_size = 0.2, random_state = 0)

but this gives me the error: Traceback (most recent call last): File "tmp.py", line 35, in <module> X_train, X_test, y_train, y_test = train_test_split(df,df_label,test_size = 0.2, random_state = 0) File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 2184, in train_test_split arrays = indexable(*arrays) File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 260, in indexable check_consistent_length(*result) File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 235, in check_consistent_length " samples: %r" % [int(l) for l in lengths]) ValueError: Found input variables with inconsistent numbers of samples: [608, 3]

How can I achieve this in python?

Use scikit learn:

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 0)

• I have already tried this. Kindly check my updated question. – yamini goel Dec 31 '18 at 11:54
• You need to have the same number of samples in both arrays, otherwise, it doesn't work. – Matthieu Brucher Dec 31 '18 at 12:47
• Matthieu is there a way to split the dataset into train and test sets when x value has a number of columns. I mean to say if I a have say input1 and input2 in x (which I want to split to x_train and x_test) and just label in y, can I somehow split it? – yamini goel Dec 31 '18 at 17:31
• yes, you can, as long as they all have the same number of entries. You can put as many vectors/list/arrays as you want, you will get the same number of output elements. – Matthieu Brucher Dec 31 '18 at 17:50
• the input1 & input2 have the shapes (3, 24, 48, 1) and y_train: (3,) When I try [input1,input2],[input1test,input2test],y_train,y_test = train_test_split([day1,day2],df_label,test_size = 0.2, random_state = 0) It gives me samples: %r" % [int(l) for l in lengths]) ValueError: Found input variables with inconsistent numbers of samples: [2, 3] error.  – yamini goel Jan 1 '19 at 4:38

It's a little unclear as to what you gain by splitting the label file into training and testing since it sounds like a reference file. Do you expect the ids to be mutually exclusive to the train vs test file? If so, that will involve a more complex, and perhaps, iterative process to get a good train / test mix.

If you don't expect the ids to be exclusive to one data set, I would just join the two data frames before performing the train/test split. In this case, your code should work as is.

In case you need it, here is a link to merging data frames.

https://pandas.pydata.org/pandas-docs/stable/merging.html

Update

The simple way is below.

X_train,X_test=train_test_split(df_label,test_size = 0.2, random_state = 0)
X_train = X_train.merge(df, left_on='id', right_on='id', how='inner')
X_test = X_test.merge(df, left_on='id', right_on='id', how='inner')


As I said though, the two data sets may be very imbalanced, but this is all I can do for you given the information you have disclosed.

Best of luck

• I want to split the label file so I could use train file for training the model and the test one so I could see the accuracy of my predictions with the actual values. I cannot have same ids in both training and testing (formed by splitting the log file)as the label file has only one value for one id. The multiple rows of a unique id has a meaning, I cannot divide them into training and testing. – yamini goel Dec 31 '18 at 17:24
• Then you should perform your split on the label file only and then join to the log file. A word of warning though, this may not produce an evenly distributed data set. – Skiddles Dec 31 '18 at 17:36
• how can I do that? Can you provide me a demo code or something? – yamini goel Dec 31 '18 at 17:38
• your code gives me this error: ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat ` – yamini goel Jan 9 '19 at 13:56
• Set the style of the column that is currently an object, to int64. – Skiddles Jan 9 '19 at 13:59