2
$\begingroup$

For what it's worth my particular case is a symmetrical matrix, but this question should be answered more generally.

$\endgroup$
  • 1
    $\begingroup$ What kind of data is it, what are you trying to optimize, how big is it, etc.? $\endgroup$ – Emre Jan 21 '16 at 3:31
  • $\begingroup$ It's basically the 0.0-1.0 similarity between every sample in a data set. The x and y axes have the ID of every sample in the same order so it ends up being a symmetrical matrix. I still feel like there's got to be a better way. $\endgroup$ – Austin Capobianco Jan 22 '16 at 22:38
2
$\begingroup$

The most compatible format is surely CSV/TSV. It's text and you can usually Gzip it on the fly with the software package you are using. There is no widely standardized format for storing matrix array data. Matlab has its *.mat files, NumPy has *.npz, Stata and SAS have their own, ... Best just use a clear-text file.

If the matrix is symmetric, if it is very large or if there will be a lot of them, you could spare 50% in space requirement by storing only the lower (or upper) triangular part of it. If you chose to do so, there is, again, no widely accepted format. Just store the shape first and then the flattened, 1D data.

$\endgroup$
2
$\begingroup$

I would go with .csv as it's universally accepted and can be read in different programming languages easily. Moreover you can simply open it with an office software. If you are using your matrix only in Python I also recommend Pickle library which writes your matrix in a .p format and can be easily read in Python with a simple load function.

$\endgroup$
1
$\begingroup$

As carriage return/line feed (CR/LF) might cause issues depending on the system, I suggest to store the matrix "inline" or in raster-style with a short header to specify your convention, a versioning number (you might change your mind later, and increase the version), etc. and at least the matrix size, and the rest in some text format (CSV,TSV). This is similar to what is done in the portable gray map or bitmap image format.

I used that to store text filter bank coefficients.

A minimum example could be: 2,3,0.1,1.2,2.3,3.4,4.5,5.6 for the $2\times 3$ matrix: \begin{array} 0.1&1.2&2.3\\3.4&4.5&5.6 \end{array} but you can use for instance #2,#3,0.1,1.2,2.3,3.4,4.5,5.6 so that aliens (think about the golden Pioneer plaque) understand that the first two integers are "different" and provide hints on how the following numbers should be read. With a square matrix (typical of symmetrical matrices), this is even more interesting, as you only need one header number #n (the side) and the acute readers will see that the remaining numbers are in $n^2$ quantity.

You can also have a look at other Matrix Storage Schemes, and if your matrix is sparse, Compressed Row Storage (CRS).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.