2
$\begingroup$

I want to create an RNN to generate a new text based on many examples of existing texts of a certain format in the training data. The type of texts in the training data consists of 3 segments, like so:

Example text 1:

[Segment 0, ~20 characters]

[Segment 1, ~200 characters]

[Segment 2, ~400 characters]

It is worth mentioning that the segments are all still alphanumerical, but of varying structure. Segment 1 contains more numbers and Segment 2 has more letters with references to occurrences in Segment 1.

I have training data with many instances of such example texts, but with varying length in each segment.

In most RNN text generation examples I have seen before the training set consists of a single long text which is split into segments of a fixed length for training, and I guess that could be done in my case as well by simply concatenating all of the different examples into one long string. Then dividing that long string into sequence pairs of "input" and "target" strings of a fixed length, say 100 characters and use them for training.

My question is if I can not instead take advantage of the fact that I have the text already split up into segments which resemble what I want to generate, and use each example as a batch for training.

In my opinion feels like it would be easier for the RNN to learn the relations between the different text sequences if it got fed data (input + target string) that was belonging to one example at a time, as opposed to data (input + target) spanning over the entire training set.

Is there any aspect that I am overlooking which would favour simply concatenating all training data into one large string?

Many thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I think that if you append a token <EOS> (end of sentence) at the end of each sentence when you merge, this would not be a problem, because the RNN would learn to cut sentences and to generate independently if you shuffle your data and train with several shuffles.

However, as you say your data is heterogeneous, you might consider to first run some clustering algorithm and then train your RNN to generate conditionally to a cluster. To build the training dataset, you can merge all data for each cluster. This is a bit more complex than simply "automatic generation" but it depends on what your goal is.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi Adam! Thanks for the reply. Do you recommend that I should apply the <EOS> token between each segment? Or between Segment 2 and next Segment 0? The clustering approach sounds interesting, do you know of an example of a similar approach I could take a look at? Thanks! $\endgroup$
    – gruffaren
    Jun 30, 2020 at 19:46
  • $\begingroup$ You append <EOS> token at the end of each segment so that when you concatenate your data the tokens are between each segment. Also training with different shuffles of the segments order can avoid some overfitting. $\endgroup$
    – Adam Oudad
    Jul 2, 2020 at 6:10
  • $\begingroup$ Concerning the clustering, I guess I would try first to embed segments in bag of word vectors, for example you apply word2vec or fasttext to each word then take the mean of all vectors in the segment. Then you can run kmeans for example (scikit-learn.org/stable/modules/clustering.html) $\endgroup$
    – Adam Oudad
    Jul 2, 2020 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.