How to make an effective sampling from a database of text documents?

Question

Problem: I want to know methods to perform an effective sampling from a database. The size of the database is about 250K text documents and in this case each text document is related to some majors (Electrical Engineering, Medicine and so on). So far I have seen some simple techniques like Simple random sample and Stratified sampling; however, I don't think it's a good idea to apply them for the following reasons:

• In the case of simple random sample, for instance, there are a few documents in the database that talk about majors like Naval Engineering or Arts. Therefore I think they are less likely to be sampled with this method but I would like to have some samples of every major as possible.

• In the case of stratified sampling, most of the documents talk about more than one major so I cannot divide the database in subgroups because they will not be mutually exclusive, at least for the case in which each major is a subgroup.

Finally, I cannot use the whole database due to expensive computational cost processing. So I would really appreciate any suggestions on other sampling methods. Thanks for any help in advance.

Answer 1

Off the top of my head, I would consider, at least, two approaches, as follows.

• Graph sampling. If you can (and it makes sense to) model your population (database of text documents) as a graph, then consider graph sampling. Check this relevant answer of mine on the topic on Cross Validated site (contains many references - I would start with this nice overview paper).

• Topic modeling. Another alternative is topic modeling (also referred to as topic models). See this introductory paper as well as this more detailed paper on the subject. Also, take a look at this interesting paper on topic modeling of streaming documents (or other situations, such as large databases, when unsupervised solutions make more sense). Finally, speaking about software for topic modeling, while many options do exist, I suggest investigating MALLET, which is an interesting Java-based software package for "statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text".

Answer 2

You're going to need to do simple random sampling, but maintain counts of the labels you've seen thus far. When you reach quota for a label, you'll need to cull all documents with that label from the pool, and sample from the remaining documents without that label.

I think this will be a fair way of getting a stratified sample when your labels are overlapping.