I want to generate documents based on a grammar to build a custom training database. What are the tools and techniques to generate random texts based on a given grammar.

More specifically, I would like to build a collection of texts with random 'sentences' and each one is a possible derivation (chosen randomly) from the grammar. I would like also to draw randomly the terminal symbols (the final words). I have looked into (python) nltk and have achieved some document generation, but the derivations chosen are not selected randomly.

  • $\begingroup$ Can you define grammar in this context? $\endgroup$ Jun 13 '16 at 11:58
  • $\begingroup$ Sure. I meant a format grammar en.wikipedia.org/wiki/Formal_grammar and more specifically a context-free grammar en.wikipedia.org/wiki/Context-free_grammar ; For example, with this nltk.org/_modules/nltk/parse/generate.html you can generate sentences from a grammar, BUT they are somewhat drawn in a graph-traversal order, I would like to generate randomly (with uniform distribution over all possible derivations, if possible) $\endgroup$
    – mic
    Jun 13 '16 at 12:03
  • $\begingroup$ Uniform distribution is a problem if the grammar defines a potentially infinite output. Often generators will limit repeated segments for this reason. $\endgroup$ Jun 13 '16 at 18:54
  • $\begingroup$ Indeed. Let's say that my grammar generates a finite number of possible derivations (including terminals which are a finite set in my case). $\endgroup$
    – mic
    Jun 14 '16 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.