Is there an existing probabilistic model that deals with the selection of subsets of words from a corpus of documents? Imagine a stack of documents where a subset of the words in each document has been highlighted. I am specifically interested in the case where the selection of words across the corpus follows a small number of patterns but the balance of each pattern is different in each document.

My data is an existing corpus of $M$ documents of varying length. For each document $m$ and word in that document $n$ I have a binary value which is 0 if the word wasn't selected and 1 if that word was selected. I know that within a document each occurrence of a word was equally likely to be selected, but that the probability that a word was selected changes between documents.

My current idea is to implement a variant of Latent Dirichlet Allocation (LDA, wikipedia). Suppose there are $K$ patterns of selecting words across the whole corpus and that the selection within each document is a mixture of each of them. Analogously to LDA, we assume that the selection of words in document $m$ is a mixture of the patterns with mixing coefficients $\theta_m\sim\operatorname{Dirichlet}$. We further assume that each pattern $k$ has a probability $\phi_{k,w}\sim\operatorname{Beta}(1,1)$ that word $w$ will be selected, which is the same across the entire corpus.

The model for selection of word $n$ in document $m$ begins by drawing $z_{m,n}\sim\operatorname{Categorical}(\theta_m)$ to determine which pattern will be used for selection. The probability that that word is selected is then distributed as $\operatorname{Bernoulli}(\phi_{z_{m,n},w_{m,n}})$, where $w_{m,n}$ returns the word at the $n$th position in document $m$. I know that this model closely mimics the true method of selection and so my question is mostly about implementation.

I know that sparse Dirichlet models like this can be hard to sample efficiently. Does anyone know of an existing implementation of this model? I can provide example data if requested!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.