Skip to main content
Chandrashekhar Patil's user avatar
Chandrashekhar Patil's user avatar
Chandrashekhar Patil's user avatar
Chandrashekhar Patil
  • Member for 5 years, 6 months
  • Last seen more than 5 years ago
  • Pune, Maharashtra, India
awarded
awarded
comment
How to deal with millions or rows of data for analysis/visualization purpose
Store data into memory. So, here memory means local hard disk memory you are telling right? So, if i get those tables available on my PC hard disk. Then do you mean, i should import them into my python notebook from hard disk n run analysis on it? I don't know, what happens behind the scene, when pandas import records or when it processes records? Where it keeps records? What is the role of RAM here n storage(hard drive) here. I am new to this,so please bear with me. Any good link, you can suggest to understand, what i mentioned above, will be greatly appreciated. Thanks a lot for giving time.
comment
How to deal with millions or rows of data for analysis/visualization purpose
Thanks Rajat. If i have good configuration of Server, then can try? Also if i create resulting combined table in Sql, then i just will import it in pandas. But still it will be a table of 6 million rows. So, can i analyse/visualize it in pandas? Or i can aggregate few rows into summary rows like sum, average n reduce rows size to less as .1 million, n then run analysis/visualization on that? Kindly suggest.
awarded
Loading…