I'm requesting data from a government body, and they asked me what format I want to receive the data in. This will be a table of about 400,000 rows and about 10 columns. My options are:

"comma or tab delimited ASCII, Microsoft Access database, Microsoft Excel file etc"

They also want to know the media by which I want to receive data. My options are:

"encrypted CD, DLT Tape, etc."

I'm used to working with JSON, and doing data analysis via Python and R. These data will have to remain confidential.

Any pointers?


Either format will work with R and Python, though you'll need a library for MS Access, which is the least common option. If any of these formats is the native format of the data, I'd go for that, which will avoid weird transformation artefacts. If neither of the formats is, I'd choose CSV, which would allow me to use a text editor if needed. However, if one of the columns contains user generated texts, it might be worth the trouble to choose Excel, that would limit the chance that the export would get botched.

It is way easier to get your hands on a CD-reader. Your data doesn't sound that impressively large that it warrants using tape.

  • $\begingroup$ Agreed with comment on csv vs excel. The pandas library in python can read excel files, and I’m sure R has a library capable of that as well. I would be skeptical that a csv created as part of a foia process would be properly generated. $\endgroup$
    – kbrose
    Feb 16 '18 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.