5
$\begingroup$

I'll set the question up with an example. You are analysing news coverage text data from 2014, and find that a term appears less often in the third quarter of 2014 than the final quarter (let's imagine it's the term "Christmas"). Unfortunately, there are also far less news articles in the third quarter than in the second (due to the lack of news in the summer). So how do we accurately compare the counts in each quarter? We assume that there will be a greater number of occurrences in the fourth quarter, but how much does the magnitude of this difference depend on the change in size of the underlying text?

Heap's law shows the relationship between text size and number of unique terms. It's non-linearity implies that the rate of new, unique words introduced by the text decreases as you increase the size of the text, and the proportion of the text taken up by each existing word therefore increases. This applies given documents taken from the same 'distribution' of text, in other words the underlying zipfian distribution of word ranks is identical (see wiki).

In my example above this is obviously not the case, since the underlying topics, and resultant term distribution, will change between summer and winter, especially with regards to the term "Christmas". But take the same term count but over the whole of 2013 and 2014; you would reasonably expect the general underlying term distribution to be the same in each period, so Heap's law applies, but what if the volume of text has changed? Simply normalising by the size of the text, or the number of documents, does not, as far as I can tell, account for the relative change in expected value of the term count.

I have a hunch that this might be a simple application of Heap's or Zipf's laws, but I can't see how to apply them to this particular question. Appreciate any insight.

Improve this question
$\endgroup$
2
  • $\begingroup$ For clarification, are you trying to calculate the relative importance of a word quarter to quarter, or are you trying to make a prediction about the the raw frequency in the next quarter? $\endgroup$
    – j.a.gartner
    May 13, 2015 at 18:28
  • $\begingroup$ Relative importance primarily, prediction secondary. Does it make a difference? $\endgroup$
    – christopherlovell
    May 13, 2015 at 18:33

3 Answers 3

Reset to default
6
+50
$\begingroup$

The approach suggested by j.a.gartner is good if you want to analyze the quarter-to-quarter change in frequency for a given term, considering only that term in isolation. What it doesn't do is evaluate the relative frequency compared to all other terms in the corpus. For this you could compare frequency rank for all terms quarter-to-quarter. This is the analysis used in Zipf's Law. Another advantage of this analysis is that you can test whether the generating process is or is not stationary (e.g. governed by same distribution quarter to quarter).

The advantage of comparing frequency rank is that it doesn't depend on the relative size of the corpus for each quarter (as long as they are all "large"). In simple language, the frequency rank of the word "Christmas" in the 4th quarter will probably be the same, regardless of whether the (US English news) corpus is 10,000 articles or 1,000,000 articles.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is an obvious approach that I hadn't considered, thanks. My only concern is that it would be sensitive to the pre-cleaning approach used, since stopword removal, or removal based on tf-idf weighting, remove the most common words. Small changes to the stopword dictionary could lead to wild changes in rank. It would also be difficult to communicate the magnitude of a change in rank, especially for lower rank terms with similar frequencies that could change rank significantly with a small change in frequency. $\endgroup$
    – christopherlovell
    May 14, 2015 at 6:36
  • $\begingroup$ @polyphant -- agreed. These are complications that would have to be overcome using the ranked frequency analysis. Doing both types of analysis would probably be a good idea. $\endgroup$
    – MrMeritology
    May 14, 2015 at 15:43
3
$\begingroup$

If you are looking to calculate the relative importance of a word in this scenario, you might consider doing an inverse document frequency score on a quarterly basis: tf-idf

Once you have inverse frequency scores for all the words in the corpus from quarter to quarter, you can normalize by the range of all scores, and then do a quarter to quarter comparison.

If you're trying to do a prediction, historical data from previous years would be very useful. You can find trends over the course of a year by doing a regression on historical data and normalize to the predicted document output for that year.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks, this is an interesting idea. Could you expand on what you mean by 'normalise by the range of all scores'? $\endgroup$
    – christopherlovell
    May 13, 2015 at 20:54
  • $\begingroup$ Sure thing. Your basic idf score will be of the form: idf = log_n([Number of documents in corpus]/[number of documents containing word]) This number varies with the number of documents that are produced, and as such you'll want to find the maximal sample IDF so that you aren't favoring words based on the fact that they came from a period when more documents are produced. There are more complex calculations for IDF that attempt to mitigate such factors, but by normalizing by maximum possible score, you fix idf to a 0-1 scale. $\endgroup$
    – j.a.gartner
    May 13, 2015 at 23:45
  • $\begingroup$ I have a few issues with this approach. Firstly, as mentioned by @MrMeritology, this approach is only valid when comparing counts for the same term. Secondly, tf-idf would negatively discriminate frequent terms appearing across all documents. If it's these frequent terms that you want to compare between periods then the weighted differences would be very small. It's an interesting idea, with it's own area of application, but I'm looking for something more general purpose, preferably taking in to account the properties of Heap's and Zipf's law that lead to the issue in the first place $\endgroup$
    – christopherlovell
    May 19, 2015 at 16:47
  • $\begingroup$ I would contend that if the method can be done for a single word, it's pretty trivial to run it across the entire corpus. None the less, the method proposed by @MrMeritology does extend more naturally to the entire data set. For your second point, if you have an idf score that shrinks and grow logrithmically, the linear growth of the tf term will overtake the the weighting for all but the most frequently used words (that would most likely be on a stop word list anyway). $\endgroup$
    – j.a.gartner
    May 19, 2015 at 17:35
  • $\begingroup$ @ j.a.gartner Even if you run it across all terms, you can't then compare the scores for those terms. Agree with you on the second point though, I ignored the tf part $\endgroup$
    – christopherlovell
    May 20, 2015 at 10:18
1
$\begingroup$

Both previous answers take an interesting approach, but neither really tackles exactly what I was asking; how to compare a given terms frequency over time, and compare with other terms over the same period, in a statistically rigorous way that accounts for the power law distribution of term frequencies with collection size.

The answer below is my work so far on the problem. It is not, by any means, a complete answer, however I post it here to elicit feedback and suggestions for improvement.

Take the definition of Zipf's law,

$f(r) \sim z_{max} r^{-\alpha}$

here, $r$ is the rank of a given term, $z_{max}$ is the frequency of the most frequent term, $\alpha$ is Zipf's exponent, and $f(r)$ is the frequency of the r-th ranked term.

Taking the log of both sides, you get a linear relationship between $log(f)$ and $log(r)$, with gradient $-\alpha$ and intercept $z_{max}$

$log(f) \sim log(z_{max}) - \alpha log(r)$

enter image description here

We want to find the expected frequency of a single term over a given period. First, split the period into equal sized windows. For the first window, $i$, we know the frequency of our term $f_i$, and the frequency of the largest term $z_{max,i}$ and $\alpha$. We can use these to find the rank of our term,

$r \sim \left(\frac{f_i(r)}{z_{max,i}}\right)^{-\alpha}$

This rank can then be used to calculate the 'expected' frequency for all subsequent time windows, which I'll indicate with $\hat{f}$,

$\hat{f}_{i+1}(r) \sim z_{max,i+1} r^{\alpha(i+1)} $

The ratio $\frac{\hat{f}}{f}$ represents the normalised frequency.

Some observations:

  1. For a stationary underlying distribution, this approach identifies minor deviations from the expected frequency of a term. A moving window could be used if the underlying text distribution changes dramatically, or exhibits seasonal behaviour.
  2. The initial choice of $r$ could be calculated for the entire period, rather than the first window.
  3. $\alpha$ could be calculated from the raw term and total frequency counts using heap's law (see Lu et al. 2010 for a derivation of Heap's law from Zipf's).
  4. This approach assumes a linear relationship between $f$ and $r$ in log space.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.