If I have a big dataset, say 1TB, and want to explore and analyze it in the same way I do in my Jupyter Notebook. Is there a way to work in a Jupyter Notebook with large data like this? How would you do it? AWS? How would you do it?

The idea would be to perform: 1. data exploration, 2. data cleaning, 3. descriptive analysis.

New contributor
David Soares Rodrigues is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • $\begingroup$ Please check - databricks.com Its the same notebooks but they are running on managed Spark, so you can use Pyspark to do all the above. You can use Azure or AWS as a backend. $\endgroup$ – Yaroslaw Homenko Apr 8 at 5:09