1
$\begingroup$

I have a situation where I need to propose a solution along with stack of technologies that need to be used for the below business case

Business Case: I am receiving a car manufacturing data set from various car manufacturing companies, the data set for each car model is different and have different number of attributes but I do receive them in csv files every 2 minutes. I need to choose a particular storage format so that it is easy to query by each car model and based on the model, i select, i should be able to display the different features of that car model. Is there a recommended architecture for this type of data analysis?

Improve this question
$\endgroup$
3
  • $\begingroup$ Are you familiar with pandas library in python? $\endgroup$
    – Green Falcon
    Commented Mar 27, 2018 at 4:35
  • $\begingroup$ Can you provide some more details? For example: Is the data accumulated or will you replace old data with new? How much data are we talking about coming in every 2 minutes and how much data will be accumulated (it will grow quickly with this rate)? Is it a single record coming in or multiple records? Do they change over time or do you have a limited variety of models? $\endgroup$
    – Gegenwind
    Commented Mar 27, 2018 at 8:11
  • $\begingroup$ Hi Gegewind: The data is accumulated & i receive about 4500 files every 2 minutes and each file has about 75K records. Yes, they change over time and yes its a limited variety of models. $\endgroup$
    – user47242
    Commented Mar 29, 2018 at 1:38

2 Answers 2

Reset to default
1
$\begingroup$

Given that the data model you receive is different, it makes best sense for you to use a NoSql store like mongodb.

Here is the tech stack I would use ( python ) :

  1. To receive requests with csv file, I would use flask microweb framework.
  2. I would run flask with gunicorn and gevent.
  3. For every csv file received, I would parse the rows based on varied columns and store them in a mongodb.
  4. I would write queries on the collection in mongodb to extract information.

I would be more than happy to elaborate more if required .

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks Nischal. May i know if it is good to use spark for parsing python itself? If yes, may I know the reason? My question is indirectly what steps typically should make use of spark and what steps typically make use of python and why? $\endgroup$
    – user47242
    Commented Mar 27, 2018 at 19:06
  • $\begingroup$ @NK7983 - Spark usecase is very different and it definitely does not fall in your current problem scope. SPARK has to do a lot of in memory processing which you do not require. $\endgroup$
    – Nischal Hp
    Commented Mar 28, 2018 at 12:00
1
$\begingroup$

AWS Cloud Solution Steps-

  1. Dump those files in S3
  2. Initiate lambda function (serverless) for pre-processing data according to your need
  3. Store this data into Redshift. You can create generalized schema to store car details.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.