2
$\begingroup$

I have a number of large datasets (10GBs) each with data fetched from a NoSQL database that I have remotely downloaded on my desktop. I would like to write a Python program to run some custom data analysis (plots - preferably interactive) and export custom reports in html or pdf.

I was wondering how people do the following:

1) Store the data. For the moment I have plain text files (each file has rows of a fixed number of columns - most of the data are categorical). Would it make sense to save those in some database (SQL) or hdf5? Any hints on which is preferrable?

2) Which plotting library would you propose for the graphs? I have seen about bookeh and matplotlib supports interactive widgets but I don't know what people normally use.

3) Could I export the analysis results in an IPython notebook and then in html programmatically?

$\endgroup$
1
  • $\begingroup$ Welcome to the site :) $\endgroup$
    – Dawny33
    Jan 6, 2016 at 16:31

1 Answer 1

1
$\begingroup$

1) Store the data. For the moment I have plain text files (each file has rows of a fixed number of columns - most of the data are categorical). Would it make sense to save those in some database (SQL) or hdf5? Any hints on which is preferable?

Yes, it would make sense to store in a local database, rather than using large csv/text files. As you say that the data is derived from a NoSQL source, I assume unstructured data. So, using a SQL/relational store is out of question. As you say you are using Python, I would suggest you use TinyDB, which is both light-weight and easy to handle.

2) Which plotting library would you propose for the graphs? I have seen about bookeh and matplotlib supports interactive widgets but I don't know what people normally use.

Matplotlib would be good enough. Actually, this question is more opinion-based than anything else. There are a lot of visualization libraries you can use, like Bokeh, Seaborn, etc.

3) Could I export the analysis results in an IPython notebook and then in html programmatically?

Yes, you can do the analytics directly in an Ipython notebook(Jupyter), which also supports Markdown and HTML cells.

In addition, you can also use widgets and interactive visualization with Jupyter Ipy notebooks and Matplotlib. Tutorials for the same

$\endgroup$
2
  • $\begingroup$ Thank you for the valuable answer! The data is post-processed to be structured, I will try TinyDB, never heard of it before. I read some people have been using SQLite with mixed opinions about speed. Regarding the plotting, I guess I need to have a custom program that uses IPython since I will have to internally read the data from SQL, run some standard functions that should not be exported at the report to send to my users, right? $\endgroup$
    – user90772
    Jan 6, 2016 at 16:57
  • $\begingroup$ Yeah, right! If you don't want to use Ipython, then you can write a Luigi script for the workflow, and connect the whole thing to a server for visualization! $\endgroup$
    – Dawny33
    Jan 6, 2016 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.