0
$\begingroup$

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient way of storing these matrices efficiently in terms of

  • format (e.g. pickle, parquet)
  • structure (e.g. scipy.sparse.csr_matrix, List of Lists)
$\endgroup$
2
  • 1
    $\begingroup$ It is not clear what is sparse here? >65% does not look like sparse, or is there some typo? Also images are not typically sparse data. $\endgroup$
    – Valentas
    Nov 27, 2020 at 10:33
  • $\begingroup$ Thanks, typo indeed as I was referring to above 95% sparsity. $\endgroup$
    – hH1sG0n3
    Nov 27, 2020 at 11:21

1 Answer 1

0
$\begingroup$

Sparse matrix compression techniques is a massively efficient way of storing sparse data. Scipy package has a variety of methods to address the above in scipy.sparse. However, none of these are compatible with matrix dimensions higher than 2.

I have found handy the Sparse package that supports Coordinate List compression (COO), for higher dimension matrices, as in my use case:

Sparse matrix compression with coo

#Load sequence array file
A = np.load('array.npy', allow_pickle=True)
sparsity = 1 - (np.count_nonzero(A) / A.size)
print( "Sparsity of A:%s%%" % np.round(sparsity,3))
Sparsity of A:0.996%

#Calculate coordinate list sparse array of A
S = sparse.COO(A)

# Size calculation.
print('Size of A in bytes: %s' %A.nbytes)
Size of A in bytes: 16563527400
print('Size of S in bytes: %s' %S.nbytes)
Size of S in bytes: 249330624

On disk:

array.npy --> 15.43 GB
array_after.npy  --> 16.40 MB
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.