I have a bunch of documents containing sequential data that I want to use to train a neural network with. It is as a collection of letters each about a 2-3000 characters long. My task is, given an excerpt of such a letter I want my network to output plausible text in the same style as the letter.
What would be the best way of presenting my data to the neural network?
One method which I've seen in tutorials is to concatenate all letters into one long text sequence and letting the network train on that. But then some training sequences will contain text from multiple letters. Suppose a letter is 500 characters long and the training sequence length is 150 then the last training sequence in that letter will contain 50 characters from that letter and 100 characters from the subsequent one. I suspect that would confuse the network.
Another idea would be to insert an end of letter token between each letter and then concatenate them. But again some training sequences would contain the end of letter token and training on them would perhaps confuse the network too.
A third idea would be to insert a very long sequence of padding tokens between each letter so that no real training sequence contains text from two different letters. But that seems very wasteful. For example, if my training sequences are 256 characters long I would have to insert 256 padding tokens between each letter.