My code is in jupiter notebook and my data set(400-800mb) is present on google drive/dropbox. My task is to load the csv file from another server into my jupiter notebook. How can I achieve this? Should I move my csv file to some other server?

  • 2
    $\begingroup$ Why don't you download the file first? It is not that big. Welcome to the site! $\endgroup$ – Emre Jun 16 '18 at 20:28
  • 1
    $\begingroup$ Also do you forget to do some analysis , it generally happens that People downscale the dtypes to save memory $\endgroup$ – Aditya Jun 17 '18 at 2:07

You can do this very easily with Pandas by calling read_csv() using your URL and setting chunksize to iterate over it if it is too large to fit into memory.

There is a certain overhead with loading data into Pandas, it could be 2-3× depending on the data, so 800M might well not fit into memory. You can download a subset of the data, say 10M of CSV and call methods such as memory_usage to determine how much memory you really need.


Using requests

You can read the file line by line iteratively. You don't need to store the whole dataset into memory. This will only maintain a single row in memory at a time. This is very fast and memory efficient.

import requests
from contextlib import closing
import csv

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"

with closing(requests.get(url, stream=True)) as r:
    f = (line.decode('utf-8') for line in r.iter_lines())
    reader = csv.reader(f, delimiter=',', quotechar='"')
    for row in reader:

Using pandas

You can also do this with pandas, however this will store an entire chunk of the dataset in memory. However, this library is very powerful and can make your processing way easier.

import pandas as pd

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"
df = pd.read_csv(url)

  • $\begingroup$ I tried it but i got error on line ‘for row in reader’ saying ‘iterator should return strings, not bytes (did you open the file text mode)’ $\endgroup$ – Anko6 Jun 17 '18 at 4:16
  • $\begingroup$ drive.google.com/… the path of the file $\endgroup$ – Anko6 Jun 17 '18 at 4:17
  • $\begingroup$ @Anko, Then you need to decode. I edited the answer. $\endgroup$ – JahKnows Jun 20 '18 at 2:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.