0
$\begingroup$

I was reading through all the internet and i can't find nothing similar what i am looking for, i only saw this topic for pd.DataFrame, np.ndarray and list datasets but i didn't find nothing explaining about the technique for tuples of (sample, target), in my real project i collected some text values from a sensor data, treated these values to convert them in floats and numpy arrays after and for each data i put the label values manually by hand according his class. Initially, i based myself in the MNIST example from the (mnist.pkl.gz) file where i realized that each digit(sample) has a corresponding label value for it and it was splitted in training, validation and test data (tuples), and after, each one of them was splitted in numpy.ndarray data corresponding the samples(float32) and labels(int64) and i am trying to do it through this algorithm:

import numpy as np
x_sample = np.asarray(np.random.rand(70,10),dtype=np.float32)
label = np.random.randint(low=1, high=20, size=70)
x_label = np.asarray(label,dtype=np.int64)
all_data = (x_sample, x_label)
numpy.random.shuffle(all_data)
training, validation, test = x[:80,:80], x[10:,10:], x[:10,:10]

here is the content of x_sample:

array([[0.7884381 , 0.5198139 , 0.7347043 , 0.53761774, 0.88316244,
        0.3311668 , 0.20574439, 0.61913455, 0.6435801 , 0.8443767 ],
       [0.8014303 , 0.33291578, 0.03985023, 0.35655856, 0.7356444 ,
        0.41212454, 0.7232083 , 0.28944376, 0.51724243, 0.12681097],
       [0.58953285, 0.8617872 , 0.3465464 , 0.60786986, 0.9399198 ,
        0.12155546, 0.4397815 , 0.31907335, 0.05435377, 0.66489726],
       [0.48845264, 0.07134458, 0.9049782 , 0.7069192 , 0.82500887,
        0.5373843 , 0.40122432, 0.95420086, 0.48039883, 0.84870946],
       [0.25930798, 0.54860264, 0.8818287 , 0.9652895 , 0.52591777,
        0.57611585, 0.323937  , 0.39891577, 0.7582166 , 0.82840425],
       [0.69377965, 0.33215034, 0.49125051, 0.37949413, 0.90669614,
        0.6704183 , 0.8733709 , 0.795047  , 0.6026962 , 0.1536723 ],
       [0.5060095 , 0.78271544, 0.53297824, 0.5259229 , 0.52236253,
        0.2990353 , 0.17861106, 0.13557936, 0.53062236, 0.14666797],
       [0.09100809, 0.6182393 , 0.6562244 , 0.4549399 , 0.39327073,
        0.14840797, 0.49136984, 0.15646574, 0.04235991, 0.37740666],
       [0.7602645 , 0.28826356, 0.24848387, 0.6319861 , 0.3363197 ,
        0.6742309 , 0.6677636 , 0.6242139 , 0.4008075 , 0.8143895 ],
       [0.774157  , 0.09084971, 0.09628123, 0.5094071 , 0.4558874 ,
        0.07699268, 0.59544396, 0.72742337, 0.47036827, 0.10973344],
       [0.8442708 , 0.8539798 , 0.6930992 , 0.86144143, 0.05520363,
        0.9135834 , 0.49537596, 0.93727493, 0.6726495 , 0.5948237 ],
       [0.42058083, 0.41315758, 0.5918037 , 0.85215133, 0.39489415,
        0.7502814 , 0.7983799 , 0.7365565 , 0.59074116, 0.27945325],
       [0.15837826, 0.08082869, 0.6629912 , 0.9831824 , 0.475054  ,
        0.82906026, 0.799274  , 0.47214893, 0.02537295, 0.75266844],
       [0.35026526, 0.7729089 , 0.38714933, 0.31009585, 0.07572034,
        0.7174428 , 0.355783  , 0.03436548, 0.25768897, 0.10944376],
       [0.12069583, 0.04349767, 0.69911855, 0.3075181 , 0.20550805,
        0.19009317, 0.68552905, 0.11784174, 0.35947776, 0.90379715],
       [0.8569486 , 0.58560294, 0.43834677, 0.6241065 , 0.347318  ,
        0.62067056, 0.04616955, 0.8710371 , 0.20901534, 0.10096856],
       [0.22850657, 0.84791416, 0.12616105, 0.33321378, 0.49160412,
        0.7997573 , 0.15768695, 0.03768501, 0.7989779 , 0.9099323 ],
       [0.587911  , 0.09919985, 0.26550367, 0.7728006 , 0.5916162 ,
        0.6913489 , 0.2803392 , 0.72656184, 0.46741307, 0.4699971 ],
       [0.66562   , 0.42966244, 0.31883126, 0.3816923 , 0.67420846,
        0.11109867, 0.537801  , 0.4857902 , 0.1179759 , 0.5509052 ],
       [0.45405668, 0.6940606 , 0.5440944 , 0.55702996, 0.7779726 ,
        0.23483372, 0.63747287, 0.89246833, 0.5432484 , 0.75630325],
       [0.29934302, 0.23468557, 0.367853  , 0.28165063, 0.8550132 ,
        0.39441624, 0.1952514 , 0.9589254 , 0.32275242, 0.19636863],
       [0.4425439 , 0.5695739 , 0.9982871 , 0.61821765, 0.76951075,
        0.9567146 , 0.54244703, 0.3715103 , 0.3297213 , 0.9385153 ],
       [0.77651155, 0.36030385, 0.30450577, 0.67711693, 0.472124  ,
        0.32898945, 0.3588709 , 0.13096265, 0.26165444, 0.25270692],
       [0.65593153, 0.46588856, 0.680324  , 0.74970984, 0.22794005,
        0.58211535, 0.30732408, 0.57129717, 0.26661146, 0.48667955],
       [0.16969565, 0.02687882, 0.21874492, 0.9804752 , 0.41332415,
        0.6437682 , 0.04894815, 0.28493315, 0.2448854 , 0.32068416],
       [0.70476776, 0.17518915, 0.6927798 , 0.24432452, 0.15505427,
        0.41269347, 0.83176184, 0.35453355, 0.88754696, 0.3442294 ],
       [0.66881895, 0.17971596, 0.892545  , 0.65156984, 0.11013364,
        0.24043244, 0.69743824, 0.09783129, 0.95923007, 0.03442115],
       [0.78083134, 0.28247902, 0.7305987 , 0.1131873 , 0.55043435,
        0.15949519, 0.30447116, 0.71613485, 0.7924715 , 0.6686032 ],
       [0.5089608 , 0.5116372 , 0.14532298, 0.50884354, 0.29055136,
        0.1599595 , 0.9113204 , 0.7051524 , 0.7735829 , 0.01971148],
       [0.89713794, 0.2823232 , 0.07845514, 0.07158056, 0.11072696,
        0.9572322 , 0.5594084 , 0.19810581, 0.48990282, 0.47327495],
       [0.00798263, 0.44725016, 0.5315442 , 0.5731946 , 0.49000663,
        0.858638  , 0.5146041 , 0.3686941 , 0.96673584, 0.5320245 ],
       [0.95911306, 0.8780109 , 0.28115687, 0.57740235, 0.96111155,
        0.43399045, 0.9302051 , 0.01998311, 0.98042315, 0.03036826],
       [0.69623005, 0.6690171 , 0.3695295 , 0.38374123, 0.5838195 ,
        0.0128198 , 0.5927486 , 0.32560483, 0.28221703, 0.61576754],
       [0.81438226, 0.60159254, 0.66808176, 0.98660797, 0.25056526,
        0.6556737 , 0.01063272, 0.53863955, 0.81606984, 0.96128947],
       [0.16044165, 0.68298006, 0.24635036, 0.77791494, 0.30523553,
        0.9980678 , 0.48377916, 0.62765104, 0.8911972 , 0.72327334],
       [0.39117676, 0.7902838 , 0.43824893, 0.7850334 , 0.99879324,
        0.56961024, 0.5678125 , 0.826441  , 0.49220917, 0.44785005],
       [0.5679101 , 0.5222929 , 0.81921464, 0.7460545 , 0.00562454,
        0.39807183, 0.68751055, 0.47909838, 0.2626596 , 0.05951797],
       [0.99766076, 0.04666068, 0.6716355 , 0.38554293, 0.9959416 ,
        0.72788566, 0.11516686, 0.14250875, 0.04943997, 0.09301051],
       [0.98882073, 0.71442574, 0.45905355, 0.20342587, 0.21567728,
        0.1395113 , 0.6820768 , 0.8979004 , 0.511218  , 0.33667463],
       [0.99645066, 0.28956234, 0.1150803 , 0.68405426, 0.2513287 ,
        0.29623672, 0.98523015, 0.05583357, 0.19652528, 0.34405017],
       [0.2644573 , 0.77543855, 0.16216566, 0.2274423 , 0.31647167,
        0.24133213, 0.73656774, 0.46090963, 0.58188814, 0.8706451 ],
       [0.20827995, 0.91745377, 0.05445696, 0.9574125 , 0.90011734,
        0.5319526 , 0.47427163, 0.93446016, 0.3938062 , 0.44010285],
       [0.60822165, 0.2283025 , 0.85318834, 0.33788195, 0.6987353 ,
        0.21905498, 0.6001706 , 0.9010304 , 0.84617853, 0.60441935],
       [0.3452462 , 0.5825702 , 0.10790027, 0.1084692 , 0.059622  ,
        0.22491871, 0.8170725 , 0.12231915, 0.9291059 , 0.35144126],
       [0.07449526, 0.574309  , 0.620578  , 0.46391514, 0.17793067,
        0.6956559 , 0.8544558 , 0.5135135 , 0.3806169 , 0.02865989],
       [0.71215415, 0.5529912 , 0.55244136, 0.7464259 , 0.71241885,
        0.23331422, 0.20611712, 0.6396109 , 0.35908782, 0.911574  ],
       [0.42528477, 0.9403745 , 0.41188288, 0.7581949 , 0.22565204,
        0.01175894, 0.7029682 , 0.01867293, 0.5755737 , 0.30493212],
       [0.15451883, 0.41981763, 0.70229906, 0.64226   , 0.564968  ,
        0.34692803, 0.14764656, 0.7186067 , 0.9473129 , 0.44122258],
       [0.7382431 , 0.3245974 , 0.8588635 , 0.9360499 , 0.7608771 ,
        0.8610076 , 0.04395575, 0.74972475, 0.5832249 , 0.6207939 ],
       [0.34000713, 0.31027594, 0.19668615, 0.46552724, 0.60408646,
        0.2327863 , 0.9667652 , 0.19861211, 0.7507445 , 0.71403235],
       [0.1325356 , 0.20863862, 0.83160186, 0.08252376, 0.7125552 ,
        0.50902236, 0.9612763 , 0.5662685 , 0.28266534, 0.49477586],
       [0.2699212 , 0.59537584, 0.1451615 , 0.22500473, 0.71263206,
        0.10738193, 0.8746872 , 0.7942923 , 0.4227869 , 0.04452927],
       [0.5300188 , 0.84817845, 0.04492909, 0.81542295, 0.46294856,
        0.57283497, 0.5080157 , 0.8973773 , 0.40948075, 0.71255004],
       [0.08799187, 0.99127257, 0.8163025 , 0.10036863, 0.31362548,
        0.6763547 , 0.96596265, 0.0725966 , 0.46508265, 0.3970549 ],
       [0.1880616 , 0.6650666 , 0.16885898, 0.70865464, 0.3708192 ,
        0.6123514 , 0.3048818 , 0.3273501 , 0.6678504 , 0.11055492],
       [0.8733036 , 0.71551114, 0.12790109, 0.1610602 , 0.6383525 ,
        0.2950214 , 0.4821728 , 0.64256096, 0.7342991 , 0.9972921 ],
       [0.92470443, 0.39551586, 0.7662377 , 0.18007767, 0.55820733,
        0.72086644, 0.00691299, 0.39947975, 0.2788803 , 0.74334264],
       [0.7780325 , 0.00231893, 0.51090115, 0.5552391 , 0.94878006,
        0.63986033, 0.98103434, 0.20206656, 0.12405983, 0.44066337],
       [0.30772004, 0.46042514, 0.0025973 , 0.5745653 , 0.5846814 ,
        0.6873136 , 0.7104756 , 0.07292311, 0.72386193, 0.5177137 ],
       [0.6864299 , 0.06640337, 0.526928  , 0.28873926, 0.6531487 ,
        0.91456944, 0.95334315, 0.11827379, 0.73734987, 0.7648343 ],
       [0.646109  , 0.4421339 , 0.64199674, 0.29240683, 0.87300384,
        0.25150016, 0.43710276, 0.54317516, 0.6825831 , 0.09163991],
       [0.87291414, 0.08923422, 0.35769263, 0.20595182, 0.39125943,
        0.5966625 , 0.3600931 , 0.17950672, 0.86496043, 0.83433443],
       [0.7244432 , 0.8581997 , 0.10607199, 0.068556  , 0.16808605,
        0.92939657, 0.97304624, 0.9601079 , 0.78934044, 0.29029167],
       [0.72685224, 0.51167786, 0.31778237, 0.86089075, 0.49086782,
        0.5474001 , 0.28438014, 0.19647646, 0.9865472 , 0.89918315],
       [0.5238998 , 0.07339662, 0.56543297, 0.05817381, 0.6005609 ,
        0.48152158, 0.16061448, 0.6332628 , 0.5955543 , 0.67818344],
       [0.36237043, 0.19250306, 0.6355158 , 0.10101647, 0.51368356,
        0.5765704 , 0.79841316, 0.7892311 , 0.3689255 , 0.45686293],
       [0.06613698, 0.4426072 , 0.52412254, 0.73527026, 0.5451764 ,
        0.779179  , 0.27027693, 0.1763142 , 0.44727728, 0.90590006],
       [0.545927  , 0.28778243, 0.37357196, 0.36386237, 0.29557422,
        0.39323354, 0.877103  , 0.7777442 , 0.09475847, 0.7945491 ],
       [0.97065485, 0.33011907, 0.06117621, 0.7663131 , 0.3759106 ,
        0.64299583, 0.7013361 , 0.42511478, 0.4436903 , 0.8591207 ],
       [0.31308335, 0.8133066 , 0.4313946 , 0.26456526, 0.0181353 ,
        0.9072234 , 0.585416  , 0.9869107 , 0.44579932, 0.49035138]],
      dtype=float32)

and x_label:

array([13, 16, 12, 16, 11, 14, 16,  4,  6, 16,  8, 11, 13, 17, 14, 11,  8,
        7, 14, 15, 12, 18, 15,  9,  4, 12,  9, 11, 17,  8,  5,  6, 18,  5,
       13,  3, 17, 16,  4,  2,  2,  9, 19, 19,  3,  7,  4, 10, 14,  1, 16,
        2,  7, 10, 10,  7,  4, 17,  4, 10, 18, 19,  2,  9, 10,  9, 17, 18,
       18, 11], dtype=int64)

But it's giving to me this error: TypeError: 'tuple' object does not support item assignment

I already tried through np.split:

training, validation, test = np.split(all_data.sample(frac=1), [int(.6*len(all_data)), int(.8*len(all_data))]) 

and another error was displayed:

AttributeError: 'tuple' object has no attribute 'sample'

I am trying to split it in 60% training, 20% validation and 20% test. This example isn't about my real project, but it is related the same idea, the dataset from my project is very large to put here, is there any suggestion that i could do to split tuples of related data(input and target) into these sets?

$\endgroup$
0
$\begingroup$

I would suggest using the well-known and tested train_test_split function from sklearn. Here is the documentation.

You start with arrays in your example, so no need to use tuples.

Because you want train, test and validation sets, you will need to split the data twice though. Here is an example, where I split the into train_and_val and test, then split the train_and_val part into (final) train and val. The test remains from the first split to leave you with a final 60-20-20 split:

from sklearn.model_selection import train_test_split

# First split 80-20 - using your original data: x_sample and label
X_train_and_val, X_test, y_train_and_val, y_test = train_test_split(
     x_sample, label, test_size=0.2, random_state=666)

To split the 80 from above into 60-20 of the entire dataset, we need to use a test_size of $20 / (60 + 20) = 0.25$.

X_train, X_val, y_train, y_val = train_test_split(
     X_train_and_val, y_train_and_val, test_size=0.25, random_state=666)

Now you have X_train, y_train (60%), X_val, y_val (20%) and X_test, y_test (60%).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.