Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory.
dataframe_chunk = pd.read_sql("SELECT * from table", con = credentials)
How do I load and store the table in a format such that I can run all pandas/numpy functions for data analysis? What is the general approach taken?
You can try sampling the data, chunking, or using spark.
This thread appears to be a similar post with additional insight into the techniques for reading/writing big datasets.
