The MNIST handwritten digit dataset uses a file format .idx. What are the advantages of this file format over alternatives such as CSV, TSX and ODS?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Generally, you will find datasets being distributed in CSV format for their simplicity and human readable format that you could ingest in any programming language with just the packages that the language is shipped with. Usually, tabular data is exported in CSV format and that is one of the reasons why MNIST dataset is not provided in CSV format.
Here is the quote from LeCun's website for storing the dataset in idx format.
The data is stored in a very simple file format designed for storing vectors and multidimensional matrices.
In terms of performance, binary file formats fairs better compared to text file formats like CSV or rich text format like ODS.
Following are some of the binary file formats that are widely used.
- Avro format
- Parquet format
- Optimized Row Columnar (ORC)
- Protocol Buffers (protobuf)
These file formats support data compression, stores data type metadata to serialize and de-serialize data effeciently.
