5
$\begingroup$

I was curious to know if shuffling ML training data is beneficial to better results?

Sorry not a lot of wisdom here, but I have been reading a post from pythonprogramming.net for this topic.

I copied this function from the post and modified to just save my shuffled data to csv file.

def Randomizing():
    df2 = df.reindex(np.random.permutation(df.index))
    df2.to_csv('C:\\Users\\Machine-Learning-Electric-Data\\randomized.csv')

Randomizing()

What appears to happen is only the index gets shuffled and all other data stays the same. I have many columns in my pd dataframe where I would need to keep all rows the same. (randomly shuffle all rows, its time series data) If this is beneficial can someone give me a tip on how to randomly shuffle my data more than just the index?

$\endgroup$
  • 1
    $\begingroup$ this question could easily be googled... one convenient way is df2.sample(frac=1.0) $\endgroup$ – oW_ Feb 15 '19 at 21:00
  • $\begingroup$ Thanks for the tips, I am running a ML regression experiment and shuffling the data cuts the rmse in half $\endgroup$ – HenryHub Feb 15 '19 at 21:05
5
$\begingroup$

Shuffling the training data is generally good practice during initial preprocessing steps.

When you do a normal train_test_split, where you'll have a 75% / 25% split, your split may overlook class ordering in the original data set. For example, class labels that might resemble a data set similar to the iris data set would include target variables that resemble the following:

For example: [0, 0, 0, 1, 2, 2, 2, 3, 3, 3, 3, 3]

You could see from this example above, that splitting your data without shuffling might lead to very poor performance in your test set evaluation. Said another way, you may only capture the classes 0, 1, and 2 in your training data and only 3 will be represented in your test data. Specifically for classification tasks, but also for other ML tasks it may be useful to shuffle your data. However, each situation is different so the best idea would be to try it both ways to see whether you see a significant improvement or not.

Hope this answers your question. Drop a comment if you would like any further clarification.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.