Why Heaps' Law Equation looks so different in this NLP course?

Question

I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed to find an answer and also the Linguistics Beta Stack Exchange community also doesn't seem to be able to help. If it's not allowed here please close it.

Heaps' Law basically is an empirical function that says the number of distinct words you'll find in a document grows as a function to the length of the document. The equation given in the Wikipedia link is

$$V_R(n) = Kn^\beta$$

where $V_R$ is the number of distinct words in a document of size $n$, and $K$ and $\beta$ are free parameters that are chosen empirically (usually $0 \le K \le 100$ and $0.4 \le \beta \le 0.6$).

I'm currently following a course on Youtube called Deep Learning for NLP by Oxford University and DeepMind. There is a slide in a lecture that demonstrates Heaps' Law in a rather different way:

The equation given with the logarithms apparently is also Heaps' Law. The fastest growing curve is a corpus for Twitter data and the slowest is for the Wall Street Journal. Tweets usually have less structure and more spelling errors, etc. compared to the WSJ which would explain the faster-growing curve.

The main question that I had is how Heaps' Law seems to have taken on the form that the author has given? It's a bit of a reach but the author didn't specify what any of these parameters are (i.e. $C$, $\alpha$, $r(w)$, $b$) and I was wondering if anybody might be familiar with Heaps' Law to give me some advise on how to solve my question.

Answer 1

The plot shows Heaps' Law but the formula is something different, it is Zipf's Law.

$f(w)$ is the relative frequency (or probability) of word $w$. That is, given a random word, it will be $w$ with probability $f(w)$. Therefore, if a document has $n$ words, it has on average $n\times f(w)$ occurrences of word $w$.

The formula can be re-written as follows: $$f(w)=C(r(w)-b)^{-\alpha}$$ which is a power-law distribution that shows Zipf's Law, however with a slightly different parameterization by introducing cut-off $b$.

1. $r(w)$ denotes the rank of word $w$. For example, if we sort all the words in a news corpus based on their frequency, $r(\text{'the'})$ would be 1, $r(\text{'be'})$ would be 2, and so on,

2. Cut-off $b$ ignores highly frequent words $r(w) \le b$, effectively shifting up the rank of remaining words,

3. $C$ is the normalizing constant, i.e. $C=\sum_{r=\left \lfloor b \right \rfloor + 1}^{\infty}(r-b)^{-\alpha}$, which gives $\sum_{w,r(w)>b} f(w) = 1$, and

4. Exponent $\alpha$ denotes the rate of drop in probability when rank increases. Higher $\alpha$, faster drop.

Exponent $\alpha$ is determined by fitting the formula to some corpus, as shown in the table. Generally, lower $\alpha$ (in the case of twitter), thus slower drop, means corpus has more word diversity.