I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was obtained as 20GB gzipped CSV files from energy meters, time series data at several seconds intervals.
data_root = r"/media/usr/USB STICK"
fname = r"meters001-050-timestamps.csv.gz"
this_file = os.path.join(data_root,fname)
assert os.path.exists(this_file), this_file
Create a chunk iterator directly over the gzip file (do not unzip!)
cols_to_keep = [0,1,2,3,7]
column_names = ['METERID','TSTAMP','ENERGY','POWER_ALL','ENERGY_OUT',]
parse_dates = ['TSTAMP']
df_iterator = pd.read_csv(this_file,
names = column_names,
Iterate over the chunks
new_df = pd.DataFrame()
count = 0
for df in df_iterator:
chunk_df_15min = df.resample('15T').first()
#chunk_df_30min = df.resample('30T').first()
#chunk_df_hourly = df.resample('H').first()
this_df = chunk_df_15min
this_df = this_df.pipe(lambda x: x[x.METERID == 1])
new_df = pd.concat([new_df,chunk_df_15min])
print("chunk",count, len(chunk_df_15min), 'rows added')
#print("chunk",i, len(temp_df),'rows added')
count += 1
Inside the chunk loop, I am doing some filtering and re-sampling on time. Doing this I reduced the size from 20GB to a few hundred MB HDF5 for further offline data exploration.