Text generation using Tensor Factorization

Question

Text generation is well studied using Markov chains or NNs, but I am not aware of any works to word sequence prediction in terms of subspace learning.

Treating phrases or sentences as temporal data such as time series, it is possible to represent word sequences as a tensor $T = WS \times W \times K$, where $WS$ is the set of word sequences present in the corpus, $W$ represents the set of segmented words, and $K$ is the maximum length of observed sequences For instance, for a phrase, ws = word sequence prediction, then $T(ws, ``sequence", 2) = 1$

For an incomplete tensor, where entries s.t. prediction are missing, the reconstructed tensor after decomposition can then be used to generate texts, in terms of the observed word space.

My questions are as follows:

1) Is there any works using tensor factorization or factorization machines for word sequence generation?

2) How subspace learning models differ from those generative models, such as Recurrent Neural Networks or Belief Networks? What are the downsides of using subspace methods as compared to other established methods?

2) How to establish the threshold for the length of the predicted sequence? For example, can one look at the $WS_r \times K_r$ space, and use cross-validation to find the threshold for each word sequences?

Any pointers or answers to any of the above questions is highly appreciated.

Answer 1

Tensor Factorization would not work for text generation as a stand-alone technique. There is no way for the decomposition to model long-term dependencies in language. Without modeling long-term language dependencies, its results would be similar to low-order Markov chains.

Tensor Factorization could be used as another signal in a larger natural language generation system, for example improving word-embeddings.