Text generation is well studied using Markov chains or NNs, but I am not aware of any works to word sequence prediction in terms of subspace learning.

Treating phrases or sentences as temporal data such as time series, it is possible to represent word sequences as a tensor $T = WS \times W \times K$, where $WS$ is the set of word sequences present in the corpus, $W$ represents the set of segmented words, and $K$ is the maximum length of observed sequences For instance, for a phrase, ws = word sequence prediction, then $T(ws, ``sequence", 2) = 1$

For an incomplete tensor, where entries s.t. prediction are missing, the reconstructed tensor after decomposition can then be used to generate texts, in terms of the observed word space.

My questions are as follows:

1) Is there any works using tensor factorization or factorization machines for word sequence generation?

2) How subspace learning models differ from those generative models, such as Recurrent Neural Networks or Belief Networks? What are the downsides of using subspace methods as compared to other established methods?

2) How to establish the threshold for the length of the predicted sequence? For example, can one look at the $WS_r \times K_r$ space, and use cross-validation to find the threshold for each word sequences?

Any pointers or answers to any of the above questions is highly appreciated.


1 Answer 1


Tensor Factorization would not work for text generation as a stand-alone technique. There is no way for the decomposition to model long-term dependencies in language. Without modeling long-term language dependencies, its results would be similar to low-order Markov chains.

Tensor Factorization could be used as another signal in a larger natural language generation system, for example improving word-embeddings.

  • $\begingroup$ May be a way to model long term dependencies is to impose an extra constraint on sequence (temporal) dimension, where each factor for this dimension is some linear transformation of its predecessor(s). Then such factor model entails the dependencies along the temporal dimension and may have generative power. $\endgroup$
    – Amir
    Oct 4, 2017 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.