My code is in jupiter notebook and my data set(400-800mb) is present on google drive/dropbox. My task is to load the csv file from another server into my jupiter notebook. How can I achieve this? Should I move my csv file to some other server?
There is a certain overhead with loading data into Pandas, it could be 2-3× depending on the data, so 800M might well not fit into memory. You can download a subset of the data, say 10M of CSV and call methods such as memory_usage to determine how much memory you really need.
You can read the file line by line iteratively. You don't need to store the whole dataset into memory. This will only maintain a single row in memory at a time. This is very fast and memory efficient.
import requests from contextlib import closing import csv url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv" with closing(requests.get(url, stream=True)) as r: f = (line.decode('utf-8') for line in r.iter_lines()) reader = csv.reader(f, delimiter=',', quotechar='"') for row in reader: print(row)
You can also do this with pandas, however this will store an entire chunk of the dataset in memory. However, this library is very powerful and can make your processing way easier.
import pandas as pd url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv" df = pd.read_csv(url) print(df.head())