0
$\begingroup$

I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory.

dataframe_chunk = pd.read_sql("SELECT * from table", con = credentials)

How do I load and store the table in a format such that I can run all pandas/numpy functions for data analysis? What is the general approach taken?

$\endgroup$

1 Answer 1

1
$\begingroup$

You can try sampling the data, chunking, or using spark.

This thread appears to be a similar post with additional insight into the techniques for reading/writing big datasets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.