3
$\begingroup$

I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats.

I need a datastore that can rapidly serve these up in batches for PyTorch training. I also need to be able to rapidly write new sequences to the store. It's like an experience replay buffer for reinforcement learning, but I want to store every single run.

These sequences should each have some attached structured metadata in a queryable format so that I can select subsets of sequences.

The best solution looks like HDF5 - either through h5py or PyTables - except that I don't know how to make it efficiently handle the variable sequence lengths. Padding isn't appropriate because of the wildly varying lengths, and storing each sequence as its own HDF5 dataset seems like a poor idea as HDF5 doesn't seem to be optimised for massive numbers of small datasets.

Ideas on my radar include Pandas multi-indexing, HDF5 region references, and building a custom metadata index system from scratch. I'm not really sure where to go from here.

Storage compactness matters - I need to be reasonably efficient with my storage space.

Improve this question
$\endgroup$
1
  • $\begingroup$ I rewrote this entire question to be more focused. $\endgroup$
    – Sam
    Commented May 5, 2020 at 6:45

1 Answer 1

Reset to default
1
$\begingroup$

What I've opted for at the moment is packing all of the samples into a single HDF5 table buffer, and keeping a separate table with metadata that tracks each individual sequence's buffer position and length. This works, but I won't be marking this answer as correct because I'm not satisfied with it. This storage method is very poorly suited to editing, and it's vulnerable to loss if a bug were to cause the tables to become out of sync.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I am wondering the same thing as you, and I also have opted for hdf5 but it leads to surprisingly large sizes. How did you structure your file? $\endgroup$
    – qmeeus
    Commented Jul 10, 2020 at 10:05
  • $\begingroup$ @qmeeus What I ended up with was a number of tables for different size ranges. Basically I had a short sequences table, a medium length sequences table, and a long sequences table. Actually I think I had five tables. I put each sequence in the smallest table that would fit it, then padded it and stored the true length. I then did some fiddly work with a custom PyTorch dataloader to sample from multiple tables in a background process. I was never happy with this system but it was a bodge that worked well enough for me to move on with the project. $\endgroup$
    – Sam
    Commented Jul 11, 2020 at 14:05
  • $\begingroup$ Thanks for your answer :) $\endgroup$
    – qmeeus
    Commented Jul 13, 2020 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.