Appending to numpy array for creating dataset

Question

I want to create a dataset from three numpy matrices - train1 = (204,), train2 = (204,) and train3 = (204,). Basically all sets are of same length. I am applying a sliding window function on each of window 4. Each set become of shape =(201,4) I want a new array in which all these values are appended row wise. Like for first train1 then train2 then train3. And final output set is of size =(603,4).

This is a sliding window function which converts array of shape (204,) to (201,4)

def moving_window(x, length, step=1):
    streams = it.tee(x, length) 
    return zip(*[it.islice(stream, i, None, step) for stream, i in zip(streams, it.count(step=step))]) 

Create dataset fucntion is:

def create_dataset(dataset1,dataset2):
    dataX=[]       
    x=list(moving_window(dataset1,4))
    x=np.asarray(x) 
    dataX.append(x)
    y=list(moving_window(dataset2,4)) 
    y=np.asarray(y) 
    dataX.append(y) 
    return np.array(dataX)

data_new=create_dataset(train1,train2)

It is returning a dataset of shape 0(2,201,4). I think this is appending differently, but I want row wise appending. so that the new _dataset is of shape= (402,4) with two sets and (603,4) with three sets. I want to generalize as well like if I want for 10 training sets or twenty training sets. How can I do that?

Answer 1

I think its because of the way you are appending the datasets in list and then converting it to numpy array.

Solution 1

One quick solution is to reshape your array as -

data_new = data_new.reshape(data_new.shape[0]*data_new.shape[1], data_new.shape[2])

So, your data of shape (2,201,4) will become (2*201,4) = (402,4).

Solution 2

Another solution is to append the arrays in the function you have defined, instead of returning np.array(dataX), use -

return np.append(x, y, axis = 0)

So, you don't have to use dataX anywhere.