Best way to handle padding in time series data such as text

Question

I have a bunch of documents containing sequential data that I want to use to train a neural network with. It is as a collection of letters each about a 2-3000 characters long. My task is, given an excerpt of such a letter I want my network to output plausible text in the same style as the letter.

What would be the best way of presenting my data to the neural network?

One method which I've seen in tutorials is to concatenate all letters into one long text sequence and letting the network train on that. But then some training sequences will contain text from multiple letters. Suppose a letter is 500 characters long and the training sequence length is 150 then the last training sequence in that letter will contain 50 characters from that letter and 100 characters from the subsequent one. I suspect that would confuse the network.

Another idea would be to insert an end of letter token between each letter and then concatenate them. But again some training sequences would contain the end of letter token and training on them would perhaps confuse the network too.

A third idea would be to insert a very long sequence of padding tokens between each letter so that no real training sequence contains text from two different letters. But that seems very wasteful. For example, if my training sequences are 256 characters long I would have to insert 256 padding tokens between each letter.

Answer 1

Time series Analysis refers to the type of problems where we have to analyse an outcome based on time dependent inputs. Time series data is basically a sequence of data, hence time series problems are often referred to as sequence problems. Recurrent Neural Networks (RNN) have been proven to efficiently solve sequence problems. Particularly, Long Short Term Memory Network (LSTM), which is a variation of RNN, is currently being used in a variety of domains to solve sequence problems.

If we are trying to obtain more features, it is a good idea to utilise the idea of distributed representation of words, in which each input is represented by many features and each feature is involved in many possible inputs. In distributional semantics, word embedding models are used to map from the one-hot vector space to a continuous vector space in a much lower dimension than conventional bag-of-words model. we can learn the dependencies between characters and the conditional probabilities of characters in sequences so that we can in turn generate wholly new and original sequences of characters.

In such a perspective, Word2Vec word embedding model would be a good choice. It could be a good idea to convert the words into vectors using word embedding models. Then the word sequences in sentences can be feed as input to the LSTM to learn the long distance contextual dependency among the words.

Typically, sequence problems can be broadly classified into one-to-one, many-to-one, one-to-many, and many-to-many types. As per the descriptions, it seems the requirement is to analyse many sequences of textual letters as different many textual sequences itself. In these sequence to sequence analysis problems, you can either choose a single feature or with multiple features.

Long Short Term Memory (LSTM) neural networks are generally suitable for sequence prediction and sequence classification problems. If we are using LSTM for this problem, we need to model the text sequence accordingly. First we need to vectorise the text by turning each text into a sequence of integers or into a vector. It is important to limit the dataset to a top word limit. Also we need to have a maximum sequence length for each text.

Then we can tokenise the text applying this model. We can truncate and pad the input sequence such that they are all same length for modelling. The sequence will not be of same length in terms of content, but same length vectors need to be constructed.