Flair Embeddings - Significance of Backwards vs Forwards?

Question

I'm working on a project that makes use of Flair for stacked embeddings. I'm looking at the built in embeddings on this page. I noticed that the table shows news-X as being "Trained with 1 billion word corpus". However when actually making use of the embeddings it seems you either use news-forward or news-backward.

I'm assuming this means both of these embeddings are trained on the same dataset and related. However one of the projects I'm looking at actually stacks both of these embeddings:

    embedding_types: List[TokenEmbeddings] = [
        FlairEmbeddings('news-forward'),
        FlairEmbeddings('news-backward'),
        # ... additional embeddings
    ]

I'm having trouble understanding what the forward and backwards means in this situation. Furthermore if they're both based on the same data what would be the benefit of stacking them?

The page linked above also says: "We recommend combining both forward and backward Flair embeddings" but doesn't explain exactly how the two different embeddings are generated from the training data.

Answer 1

Upon further research I found that during training forward language models try to predict the next word in a sequence. Backwards language models on the other hand start at the end of a sequence and attempt to predict the proceeding word. It seems that by stacking both forward and backwards models produced from the same data-set you get better results than using a forward or backwards model alone.