0
$\begingroup$

Is there an existing probabilistic model that deals with the selection of subsets of words from a corpus of documents? Imagine a stack of documents where a subset of the words in each document has been highlighted. I am specifically interested in the case where the selection of words across the corpus follows a small number of patterns but the balance of each pattern is different in each document.

My data is an existing corpus of $M$ documents of varying length. For each document $m$ and word in that document $n$ I have a binary value which is 0 if the word wasn't selected and 1 if that word was selected. I know that within a document each occurrence of a word was equally likely to be selected, but that the probability that a word was selected changes between documents.

My current idea is to implement a variant of Latent Dirichlet Allocation (LDA, wikipedia). Suppose there are $K$ patterns of selecting words across the whole corpus and that the selection within each document is a mixture of each of them. Analogously to LDA, we assume that the selection of words in document $m$ is a mixture of the patterns with mixing coefficients $\theta_m\sim\operatorname{Dirichlet}$. We further assume that each pattern $k$ has a probability $\phi_{k,w}\sim\operatorname{Beta}(1,1)$ that word $w$ will be selected, which is the same across the entire corpus.

The model for selection of word $n$ in document $m$ begins by drawing $z_{m,n}\sim\operatorname{Categorical}(\theta_m)$ to determine which pattern will be used for selection. The probability that that word is selected is then distributed as $\operatorname{Bernoulli}(\phi_{z_{m,n},w_{m,n}})$, where $w_{m,n}$ returns the word at the $n$th position in document $m$. I know that this model closely mimics the true method of selection and so my question is mostly about implementation.

I know that sparse Dirichlet models like this can be hard to sample efficiently. Does anyone know of an existing implementation of this model? I can provide example data if requested!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.