0
$\begingroup$

I have about a few million records (small CSV / JSON file) from different sources, with about 50k added everyday. All on my local host.

Until now, I have been using simple file structure to manage them, but it's getting cumbersome. Ideally, I'd like to query files by their meta data (source, type, etc), and pipe that into my ML pipeline (TFX). Id like to keep them local if possible

does anyone have a good solution that you think will work well ?

All the best!

Johnny

Improve this question
$\endgroup$
3
  • $\begingroup$ These CSV/JSON files you have, are they downloaded manually and placed in the respective folders? or are they created automatedly by a system? $\endgroup$
    – Akshay
    Commented Nov 16, 2021 at 3:28
  • $\begingroup$ Downloaded from the internet, and placed in specific path. The path & file name acts as the metadata currently, which is getting hard to manage $\endgroup$
    – Johnny
    Commented Nov 17, 2021 at 5:00
  • $\begingroup$ Since you have a large volume of data constantly added I would suggest your first step must be to try to automate this process. Prepare an ETL pipeline which regularly downloads data from APIs and dumps data to your database. Explore possibility of fetching same info from APIs instead of file downloads. $\endgroup$
    – Akshay
    Commented Nov 17, 2021 at 5:29

1 Answer 1

Reset to default
1
$\begingroup$

So after many experiments, this is what I landed with:

Raw CSV gets converted to Parquet, then Parquet gets stored into Minio.

A few considerations:

  1. Parquet files are fast and I don't have to consistently change schema in my code
  2. I can use Apache Drill to query Parquet files stored on Minio directly, and then I can use Superset to do analysis

Cheers

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.