How do you feed data from a data warehouse to Python for ad-hoc analysis?

My day-to-day work is to answer ad-hoc questions, and 95% of the data I need is in our data warehouse. I often query data from our warehouse to CSV file(s), then use Python to load these files with other sources to analyze.

I am going to work with a very gigantic data warehouse that may not be possible to use the same csv method.

Our data warehouse is in Redshift.

What is your experience to feed python/R for data analysis?

New contributor
Bunny Eat Data is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

2 Answers 2

>>> import pandas

#Connect to the cluster
>>> import redshift_connector
>>> conn = redshift_connector.connect(
# Create a Cursor object
>>> cursor = conn.cursor()

# Query and receive result set
cursor.execute("select * from book")
result: pandas.DataFrame = cursor.fetch_dataframe()

Lot's of BigData libraries out there. PySpark and Hadoop are a couple you could use (personally recommend PySpark for it's pythonic usage).

If you only want to stick to pandas (for some reason), you could sample the data for initial anaylsis. Then increase the sample size gradually in iterations till you memory size is reached. Obviously this technique will require more time and effort!

I would go for the first technique!



Your Answer

Bunny Eat Data is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.