# Appending to numpy array for creating dataset

I want to create a dataset from three numpy matrices - train1 = (204,), train2 = (204,) and train3 = (204,). Basically all sets are of same length. I am applying a sliding window function on each of window 4. Each set become of shape =(201,4) I want a new array in which all these values are appended row wise. Like for first train1 then train2 then train3. And final output set is of size =(603,4).

This is a sliding window function which converts array of shape (204,) to (201,4)

def moving_window(x, length, step=1):
streams = it.tee(x, length)
return zip(*[it.islice(stream, i, None, step) for stream, i in zip(streams, it.count(step=step))])


Create dataset fucntion is:

def create_dataset(dataset1,dataset2):
dataX=[]
x=list(moving_window(dataset1,4))
x=np.asarray(x)
dataX.append(x)
y=list(moving_window(dataset2,4))
y=np.asarray(y)
dataX.append(y)
return np.array(dataX)

data_new=create_dataset(train1,train2)


It is returning a dataset of shape 0(2,201,4). I think this is appending differently, but I want row wise appending. so that the new _dataset is of shape= (402,4) with two sets and (603,4) with three sets. I want to generalize as well like if I want for 10 training sets or twenty training sets. How can I do that?

• What is it.tee(x,length) function doing? I don't get this. Jul 9, 2018 at 10:12
• The tee() function returns several independent iterators (defaults to 2) based on a single original input. It has semantics similar to the Unix tee utility, which repeats the values it reads from its input and writes them to a named file and standard output. Basically this whole function changing the input of shape(204,) into (201,4) Jul 9, 2018 at 10:39

I think its because of the way you are appending the datasets in list and then converting it to numpy array.

Solution 1

One quick solution is to reshape your array as -

data_new = data_new.reshape(data_new.shape*data_new.shape, data_new.shape)

So, your data of shape (2,201,4) will become (2*201,4) = (402,4).

Solution 2

Another solution is to append the arrays in the function you have defined, instead of returning np.array(dataX), use -

return np.append(x, y, axis = 0)


So, you don't have to use dataX anywhere.

• Its working for two dataset. like if we have to do for 10 or 20 datasets? Jul 10, 2018 at 0:25
• You can use either a for loop or make your create_dataset function recursive so it takes all available data. Jul 10, 2018 at 4:34