How do you compare term counts between two different periods, with different underlying corpus sizes, without bias?

I'll set the question up with an example. You are analysing news coverage text data from 2014, and find that a term appears less often in the third quarter of 2014 than the final quarter (let's imagine it's the term "Christmas"). Unfortunately, there are also far less news articles in the third quarter than in the second (due to the lack of news in the summer). So how do we accurately compare the counts in each quarter? We assume that there will be a greater number of occurrences in the fourth quarter, but how much does the magnitude of this difference depend on the change in size of the underlying text?

Heap's law shows the relationship between text size and number of unique terms. It's non-linearity implies that the rate of new, unique words introduced by the text decreases as you increase the size of the text, and the proportion of the text taken up by each existing word therefore increases. This applies given documents taken from the same 'distribution' of text, in other words the underlying zipfian distribution of word ranks is identical (see wiki).

In my example above this is obviously not the case, since the underlying topics, and resultant term distribution, will change between summer and winter, especially with regards to the term "Christmas". But take the same term count but over the whole of 2013 and 2014; you would reasonably expect the general underlying term distribution to be the same in each period, so Heap's law applies, but what if the volume of text has changed? Simply normalising by the size of the text, or the number of documents, does not, as far as I can tell, account for the relative change in expected value of the term count.

I have a hunch that this might be a simple application of Heap's or Zipf's laws, but I can't see how to apply them to this particular question. Appreciate any insight.

• For clarification, are you trying to calculate the relative importance of a word quarter to quarter, or are you trying to make a prediction about the the raw frequency in the next quarter? – j.a.gartner May 13 '15 at 18:28
• Relative importance primarily, prediction secondary. Does it make a difference? – christopherlovell May 13 '15 at 18:33

The approach suggested by j.a.gartner is good if you want to analyze the quarter-to-quarter change in frequency for a given term, considering only that term in isolation. What it doesn't do is evaluate the relative frequency compared to all other terms in the corpus. For this you could compare frequency rank for all terms quarter-to-quarter. This is the analysis used in Zipf's Law. Another advantage of this analysis is that you can test whether the generating process is or is not stationary (e.g. governed by same distribution quarter to quarter).

The advantage of comparing frequency rank is that it doesn't depend on the relative size of the corpus for each quarter (as long as they are all "large"). In simple language, the frequency rank of the word "Christmas" in the 4th quarter will probably be the same, regardless of whether the (US English news) corpus is 10,000 articles or 1,000,000 articles.

• This is an obvious approach that I hadn't considered, thanks. My only concern is that it would be sensitive to the pre-cleaning approach used, since stopword removal, or removal based on tf-idf weighting, remove the most common words. Small changes to the stopword dictionary could lead to wild changes in rank. It would also be difficult to communicate the magnitude of a change in rank, especially for lower rank terms with similar frequencies that could change rank significantly with a small change in frequency. – christopherlovell May 14 '15 at 6:36
• @polyphant -- agreed. These are complications that would have to be overcome using the ranked frequency analysis. Doing both types of analysis would probably be a good idea. – MrMeritology May 14 '15 at 15:43

If you are looking to calculate the relative importance of a word in this scenario, you might consider doing an inverse document frequency score on a quarterly basis: tf-idf

Once you have inverse frequency scores for all the words in the corpus from quarter to quarter, you can normalize by the range of all scores, and then do a quarter to quarter comparison.

If you're trying to do a prediction, historical data from previous years would be very useful. You can find trends over the course of a year by doing a regression on historical data and normalize to the predicted document output for that year.

• Thanks, this is an interesting idea. Could you expand on what you mean by 'normalise by the range of all scores'? – christopherlovell May 13 '15 at 20:54
• Sure thing. Your basic idf score will be of the form: idf = log_n([Number of documents in corpus]/[number of documents containing word]) This number varies with the number of documents that are produced, and as such you'll want to find the maximal sample IDF so that you aren't favoring words based on the fact that they came from a period when more documents are produced. There are more complex calculations for IDF that attempt to mitigate such factors, but by normalizing by maximum possible score, you fix idf to a 0-1 scale. – j.a.gartner May 13 '15 at 23:45
• I have a few issues with this approach. Firstly, as mentioned by @MrMeritology, this approach is only valid when comparing counts for the same term. Secondly, tf-idf would negatively discriminate frequent terms appearing across all documents. If it's these frequent terms that you want to compare between periods then the weighted differences would be very small. It's an interesting idea, with it's own area of application, but I'm looking for something more general purpose, preferably taking in to account the properties of Heap's and Zipf's law that lead to the issue in the first place – christopherlovell May 19 '15 at 16:47
• I would contend that if the method can be done for a single word, it's pretty trivial to run it across the entire corpus. None the less, the method proposed by @MrMeritology does extend more naturally to the entire data set. For your second point, if you have an idf score that shrinks and grow logrithmically, the linear growth of the tf term will overtake the the weighting for all but the most frequently used words (that would most likely be on a stop word list anyway). – j.a.gartner May 19 '15 at 17:35
• @ j.a.gartner Even if you run it across all terms, you can't then compare the scores for those terms. Agree with you on the second point though, I ignored the tf part – christopherlovell May 20 '15 at 10:18

Both previous answers take an interesting approach, but neither really tackles exactly what I was asking; how to compare a given terms frequency over time, and compare with other terms over the same period, in a statistically rigorous way that accounts for the power law distribution of term frequencies with collection size.

The answer below is my work so far on the problem. It is not, by any means, a complete answer, however I post it here to elicit feedback and suggestions for improvement.

Take the definition of Zipf's law,

$$f(r) \sim z_{max} r^{-\alpha}$$

here, $$r$$ is the rank of a given term, $$z_{max}$$ is the frequency of the most frequent term, $$\alpha$$ is Zipf's exponent, and $$f(r)$$ is the frequency of the r-th ranked term.

Taking the log of both sides, you get a linear relationship between $$log(f)$$ and $$log(r)$$, with gradient $$-\alpha$$ and intercept $$z_{max}$$

$$log(f) \sim log(z_{max}) - \alpha log(r)$$ We want to find the expected frequency of a single term over a given period. First, split the period into equal sized windows. For the first window, $$i$$, we know the frequency of our term $$f_i$$, and the frequency of the largest term $$z_{max,i}$$ and $$\alpha$$. We can use these to find the rank of our term,

$$r \sim \left(\frac{f_i(r)}{z_{max,i}}\right)^{-\alpha}$$

This rank can then be used to calculate the 'expected' frequency for all subsequent time windows, which I'll indicate with $$\hat{f}$$,

$$\hat{f}_{i+1}(r) \sim z_{max,i+1} r^{\alpha(i+1)}$$

The ratio $$\frac{\hat{f}}{f}$$ represents the normalised frequency.

Some observations:

1. For a stationary underlying distribution, this approach identifies minor deviations from the expected frequency of a term. A moving window could be used if the underlying text distribution changes dramatically, or exhibits seasonal behaviour.
2. The initial choice of $$r$$ could be calculated for the entire period, rather than the first window.
3. $$\alpha$$ could be calculated from the raw term and total frequency counts using heap's law (see Lu et al. 2010 for a derivation of Heap's law from Zipf's).
4. This approach assumes a linear relationship between $$f$$ and $$r$$ in log space.