0
$\begingroup$

I'm the product manager for an online app. I'm currently researching a new feature where our users will be able to access "all their raw application data". This data is likely to be used by data scientists, it's also likely to be loaded into BI tools. The datasets will contain up to a few million rows.

How should I actually expose the data in a practical sense?

For example:

  • Online datasource like Amazon Redshift
  • Some other RDBMS available online (e.g. a dedicated postgres installation)
  • CSV files available on S3
  • CSV files available for download in a web interface
  • Dumps into Google sheets
  • Doesn't matter as decent data scientists can easily handle and automate anything
$\endgroup$
  • $\begingroup$ There's no definite answer to this but I'd prefer something simple like a Web-API that returns Json or CSV (depending on the structure of the data). If I can curl it it's simple enough. $\endgroup$ – stmax Mar 21 '17 at 8:21
0
$\begingroup$

Usually, people are indeed quite indifferent to the format, as long as there is an easy way in which they can transfer it to their favorite format.

CSV is a very common format, since most tools can load it. Note that there are some dataset whose CSV representation is inconvenient (e.g., the text has many commas, data set in which the type is importnat yet hard to deduce).

As for the storage of the format most people are even more indifferent to the source too. Web interface is the common option.

If you use a relational database you gain few advantages:

  • The dataset is structured so you are protected from wrong structuring problems.
  • The dataset can be naturally updated (e.g., you can keep appending records to it on a daily basis).
  • You can allow the users load only part of the dataset, which is convenient with large dataset (e.g., give me just America users from the last month).
  • In case that you are willing to enable that, you can let the users work on your database directly (e.g., as Google Big Query host github data)

To summarize, the location of the information is not important. Unless you need one of the advantages provided by a database, use a CSV.

Need a database:

  • Online datasource like Amazon Redshift
  • Some other RDBMS available online (e.g. a dedicated postgres installation)
  • Big query

The raw data is enough

  • CSV files available on S3

  • CSV files available for download in a web interface

  • Dumps into Google sheets

General true

  • Doesn't matter as decent data scientists can easily handle and automate anything (anything reasonable but yet, it is nice that you are looking for the most convenient format).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.