I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will use the website that I build and the API calls will be made which will run some machine learning algorithms on the dataset. Now I don't have much knowledge regarding designing such systems. One possible solution that I have in mind is.

Migrate the csv data to a proper database and then create an ETL system to access the data and proceed with the computations. Since I don't have much knowledge in this regard can anyone provide help on how I could create an architecture for the task at hand.

  • $\begingroup$ Welcome to DataScienceSE. Yes, you should definitely store the data in a database. Be careful, what you're planning to do is not easy at all, especially with huge data like this. You're aware that after 1 year of collecting 30GB a day you will need more than 10 TB storage just to store the data. And if you're planning to use this data for ML, you're also going to need massive computational power. $\endgroup$ – Erwan Apr 8 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.