2
$\begingroup$

Can anyone recommend any command line tool for converting large CSV file into HDF5 format?

$\endgroup$
  • 2
    $\begingroup$ depending on your definition of "large", you can use: python -c "import pandas as pd; pd.read_csv('input_file.csv').to_hdf('output_file.hdf5', key='data')" $\endgroup$ – louic Jun 3 at 12:46
0
$\begingroup$
import numpy as np
import pandas as pd

#filename = '/tmp/test.hdf5'
filename = 'D:\test.hdf5'

df = pd.DataFrame(np.arange(10).reshape((4,2)), columns=['C1', 'C2'])
print(df)
#    C1  C2
# 0  0   1
# 1  2   3
# 2  4   5
# 3  6   7

# Save to HDF5
df.to_hdf(filename, 'data', mode='w', format='table')
del df    # allow df to be garbage collected

# Append more data
df2 = pd.DataFrame(np.arange(10).reshape((4,2))*10, columns=['C1', 'C2'])
df2.to_hdf(filename, 'data', append=True)

print(pd.read_hdf(filename, 'data'))

  • 2nd approach: you could append to a HDFStore instead of calling df.to_hdf:
import numpy as np
import pandas as pd

#filename = '/tmp/test.hdf5'
filename = 'D:\test.hdf5'
store = pd.HDFStore(filename)

for i in range(2):
    df = pd.DataFrame(np.arange(10).reshape((4,2)) * 10**i, columns=['C1', 'C2'])
    store.append('data', df)

store.close()

store = pd.HDFStore(filename)
data = store['data']
print(data)
store.close()
  • 3rd approach: using chunksize parameter and append each chunk to the HDF file which was answered here.

Personally I like 1st and 2nd approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.