1
$\begingroup$

I want to create a dataset from three numpy matrices - train1 = (204,), train2 = (204,) and train3 = (204,). Basically all sets are of same length. I am applying a sliding window function on each of window 4. Each set become of shape =(201,4) I want a new array in which all these values are appended row wise. Like for first train1 then train2 then train3. And final output set is of size =(603,4).

This is a sliding window function which converts array of shape (204,) to (201,4)

def moving_window(x, length, step=1):
    streams = it.tee(x, length) 
    return zip(*[it.islice(stream, i, None, step) for stream, i in zip(streams, it.count(step=step))]) 

Create dataset fucntion is:

def create_dataset(dataset1,dataset2):
    dataX=[]       
    x=list(moving_window(dataset1,4))
    x=np.asarray(x) 
    dataX.append(x)
    y=list(moving_window(dataset2,4)) 
    y=np.asarray(y) 
    dataX.append(y) 
    return np.array(dataX)

data_new=create_dataset(train1,train2)

It is returning a dataset of shape 0(2,201,4). I think this is appending differently, but I want row wise appending. so that the new _dataset is of shape= (402,4) with two sets and (603,4) with three sets. I want to generalize as well like if I want for 10 training sets or twenty training sets. How can I do that?

$\endgroup$
  • $\begingroup$ What is it.tee(x,length) function doing? I don't get this. $\endgroup$ – Ankit Seth Jul 9 '18 at 10:12
  • $\begingroup$ The tee() function returns several independent iterators (defaults to 2) based on a single original input. It has semantics similar to the Unix tee utility, which repeats the values it reads from its input and writes them to a named file and standard output. Basically this whole function changing the input of shape(204,) into (201,4) $\endgroup$ – Hazel Jul 9 '18 at 10:39
0
$\begingroup$

I think its because of the way you are appending the datasets in list and then converting it to numpy array.

Solution 1

One quick solution is to reshape your array as -

data_new = data_new.reshape(data_new.shape[0]*data_new.shape[1], data_new.shape[2])

So, your data of shape (2,201,4) will become (2*201,4) = (402,4).

Solution 2

Another solution is to append the arrays in the function you have defined, instead of returning np.array(dataX), use -

return np.append(x, y, axis = 0)

So, you don't have to use dataX anywhere.

$\endgroup$
  • $\begingroup$ Its working for two dataset. like if we have to do for 10 or 20 datasets? $\endgroup$ – Hazel Jul 10 '18 at 0:25
  • $\begingroup$ You can use either a for loop or make your create_dataset function recursive so it takes all available data. $\endgroup$ – Ankit Seth Jul 10 '18 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.