0
$\begingroup$

I have two files namely - log.csv and label.csv. My aim is to to split these files into training and testing datasets before preprocessing the log_train file further.

The log.csv file has a no. of columns including id. For one value of id, the file has 'z' no. of rows (this 'z' is different for different value of id) . Whereas, in the label.csv file there is just one row for each value of id. I wish to split the files into - log_train.csv, log_test.csv, label_train.csv and label_test.csv obviously such that all rows corresponding to one value of id goes either to train or test file with corresponding values in label_train or label_test file.

I have already tried doing: X_train,X_test,y_train,y_test=train_test_split(df,df_label,test_size = 0.2, random_state = 0)

but this gives me the error: Traceback (most recent call last): File "tmp.py", line 35, in <module> X_train, X_test, y_train, y_test = train_test_split(df,df_label,test_size = 0.2, random_state = 0) File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 2184, in train_test_split arrays = indexable(*arrays) File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 260, in indexable check_consistent_length(*result) File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 235, in check_consistent_length " samples: %r" % [int(l) for l in lengths]) ValueError: Found input variables with inconsistent numbers of samples: [608, 3]

How can I achieve this in python?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Use scikit learn:

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 0)
Improve this answer
$\endgroup$
6
  • $\begingroup$ I have already tried this. Kindly check my updated question. $\endgroup$
    – yamini goel
    Dec 31, 2018 at 11:54
  • $\begingroup$ You need to have the same number of samples in both arrays, otherwise, it doesn't work. $\endgroup$
    – Matthieu Brucher
    Dec 31, 2018 at 12:47
  • $\begingroup$ Matthieu is there a way to split the dataset into train and test sets when x value has a number of columns. I mean to say if I a have say input1 and input2 in x (which I want to split to x_train and x_test) and just label in y, can I somehow split it? $\endgroup$
    – yamini goel
    Dec 31, 2018 at 17:31
  • $\begingroup$ yes, you can, as long as they all have the same number of entries. You can put as many vectors/list/arrays as you want, you will get the same number of output elements. $\endgroup$
    – Matthieu Brucher
    Dec 31, 2018 at 17:50
  • $\begingroup$ the input1 & input2 have the shapes (3, 24, 48, 1) and y_train: (3,) When I try [input1,input2],[input1test,input2test],y_train,y_test = train_test_split([day1,day2],df_label,test_size = 0.2, random_state = 0) It gives me samples: %r" % [int(l) for l in lengths]) ValueError: Found input variables with inconsistent numbers of samples: [2, 3] error. ` $\endgroup$
    – yamini goel
    Jan 1, 2019 at 4:38
0
$\begingroup$

It's a little unclear as to what you gain by splitting the label file into training and testing since it sounds like a reference file. Do you expect the ids to be mutually exclusive to the train vs test file? If so, that will involve a more complex, and perhaps, iterative process to get a good train / test mix.

If you don't expect the ids to be exclusive to one data set, I would just join the two data frames before performing the train/test split. In this case, your code should work as is.

In case you need it, here is a link to merging data frames.

https://pandas.pydata.org/pandas-docs/stable/merging.html

Update

The simple way is below.

X_train,X_test=train_test_split(df_label,test_size = 0.2, random_state = 0)
X_train = X_train.merge(df, left_on='id', right_on='id', how='inner')
X_test = X_test.merge(df, left_on='id', right_on='id', how='inner')

As I said though, the two data sets may be very imbalanced, but this is all I can do for you given the information you have disclosed.

Best of luck

Improve this answer
$\endgroup$
9
  • $\begingroup$ I want to split the label file so I could use train file for training the model and the test one so I could see the accuracy of my predictions with the actual values. I cannot have same ids in both training and testing (formed by splitting the log file)as the label file has only one value for one id. The multiple rows of a unique id has a meaning, I cannot divide them into training and testing. $\endgroup$
    – yamini goel
    Dec 31, 2018 at 17:24
  • $\begingroup$ Then you should perform your split on the label file only and then join to the log file. A word of warning though, this may not produce an evenly distributed data set. $\endgroup$
    – Skiddles
    Dec 31, 2018 at 17:36
  • $\begingroup$ how can I do that? Can you provide me a demo code or something? $\endgroup$
    – yamini goel
    Dec 31, 2018 at 17:38
  • $\begingroup$ your code gives me this error: ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat $\endgroup$
    – yamini goel
    Jan 9, 2019 at 13:56
  • $\begingroup$ Set the style of the column that is currently an object, to int64. $\endgroup$
    – Skiddles
    Jan 9, 2019 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.