The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to store the data.

  • 1
    $\begingroup$ If you are using pytorch, you can put your data in pt files, using torch.save and torch.load $\endgroup$ – Astariul Nov 8 '19 at 0:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.