I asked this question in the discussion section of (https://github.com/NVIDIA/Megatron-LM/issues/975) but did not receive any response so asking here with a little bit more details hoping someone had some experience with Megatron LM. I am trying to figure out if bucketization is done (or can be done) for Megatron LM model training. By "bucketization" I am referring to batching based on similar sequence length (https://torchtext.readthedocs.io/en/latest/data.html#bucketiterator).. The motivation is that I have documents with very large text and I want pick a splitting schema (which may create a lot of samples with small number of tokens and bucketize them). That brings me to the second question which is - is splitting supported in Megatron?
Any answer would be much appreciated - Thanks.
