4
$\begingroup$

I was wondering if you will be as so kind to assist me with a quick question (will to be happy to explain more if you are willing to...). I am researching and setting up a system to do a machine learning job (training) to find correlations between Social Media (or other digital trails from wearables etc.) information of a user and his scores on personality tests.

The scores are in my Postgresql (on AWS) and I need to decide on how to store the Social Media/Digital trails from wearables (unstructured and structured) information. I was thinking DynamoDB.

I was also thinking to integrate both databases under Amazon Redshift and to do the analytics (using RapidMinder) from there..... Does it all make sense? Do I really need a data warehouse for this? Will it be more sensible to use just a single DB (Postgresql or Dynamo) for all this without data warehousing? to I am talking about up to 100K records more or less (for the training).... Future data will in the millions.

I get so many conflicting answers and I hope and will appreciate your kindness and advice. Thank you so much in advance!!!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The main purpose of a datawarehouse is the ability to aggregate different types of data, and columns in a rapid way ( near realtime ) . Storage capability isn't the problem Datawarehousing is trying to solve. I can't really answer your question since i'm not quite aware of the volume of analytics you want to perform, but if it's for training a model over and over ( Online Learning ) , just set up a whole pipeline to apply the transformations you want ( the ETL part ) , model, train, and run your predictive method as many times as you want.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you Blenz for taking the time and answering. Just to be clear, assuming I will have two databases (Postgresql for the questionnaires) and DynamoDB for the messier data (from Social media and from wearables and other data) and that realtime in training and modeling is of no issue, then Data Warehouse is not needed if I understand you correctly. This brings up one final question, will it be more sensible to use only one database (the postgresql or dynamoDB) and simply to do all training and modeling from there? $\endgroup$
    – Yonathan Mizrachi
    Aug 9, 2019 at 15:02
  • $\begingroup$ If youre using data from both databases for training, you'll have to gather the incoming data in one place to do your modelling. $\endgroup$
    – Blenz
    Aug 9, 2019 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.