I've been using Python for quite some time now, scripting command line tools, etc. I'm just now moving to Data Analysis with the language; are there any tips for handling larger json files. Particularly, determining appropriate specs for a system, threading/multiprocessing, etc.

I'm finding that it's taking an excessive amount of time to handle basic tasks; I've worked with python reading and processing large files (i.e. Log files), and it seems to run a lot faster.

Example, I'm downloaded a json file from catalog.data.gov for traffic violations. The file is 758Mb in size and it takes a long time to do something very simple.

>>> import ijson
>>> filename = 'rows.json'
>>> with open(filename, 'r') as f:
...     objects = ijson.items(f, 'meta.view.columns.item')
...     columns = list(objects)
...     print columns[0]
...     f.close()

{u'name': u'sid', u'format': {}, u'dataTypeName': u'meta_data', u'fieldName': u':sid', u'renderTypeName': u'meta_data', u'position': 0, u'id': -1, u'flags': [u'hidden']}

I've worked with json before, but this seems to be taking a long time for something so simple. I'm did this on a laptop with 4GB of RAM (~2G free).

  • 1
    $\begingroup$ First try pandas.read_json. The real solution is to upgrade your computer because the file's size and your laptop's memory are considered small today. Also consider using command line tools like jq instead. Welcome to the site! $\endgroup$ – Emre Nov 23 '17 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.