1
$\begingroup$

I'm trying to do a simple reshape of a 60000,28,28 list of mnist digits into a 60000,784 numpy array where the digits have been unrolled.

To do this the code is this:

(xdata,xlabel),(ydata,ylabel)=tf.keras.datasets.mnist.load_data()
newxdata=np.array([])
cnt=0
for i in xdata:
 tmpx=i.ravel()
 if cnt == 0:
  newxdata=np.concatenate((newxdata,tmpx))
 else:
  newxdata=np.vstack((newxdata,tmpx))
 cnt=cnt+1

Why does this take so long to run? Is there a way to speed it up? Ultimately the data will be fed into a keras model in smaller batches. Would writing a generator that does the loop unrolling when a batchsize is asked for be more performant or would it not make a difference?

$\endgroup$
2
  • $\begingroup$ You are looping 60K times, that's why. You must use the available reshape method $\endgroup$
    – 10xAI
    Commented Nov 5, 2020 at 15:07
  • $\begingroup$ So i thought the concatenation operation would be O(1) in time, but in fact, there's the creation of an array the size of the previous data as a new item gets appended to it, which makes it O(n). I'm not sure what python does for reshape operations, but it is definitely faster $\endgroup$
    – trendulous
    Commented Nov 6, 2020 at 6:16

2 Answers 2

1
$\begingroup$

You can reshape a numpy array simply by:

newxdata =  xdata.reshape((60000,28*28))

for example. Or simply:

newxdata =  xdata.reshape((len(xdata),-1))

Note that reshape is a numpy function which can used also as:

import numpy as np
newxdata =  np.reshape(xdata, (60000,-1))

To speed up your loop you could alternatively use libraries like multiprocessing.Pool or CuPy or Numba.

$\endgroup$
0
$\begingroup$

I went with that suggestion and had a few additions to the implementation.

Ultimately the newly transformed data would be ingested by a keras function that requires an input of ( batch_size, unrolled image), so i created a generator function .returnbatch(batch_size) that returns a ( batchsize, unrolledimage )

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.