# How to do vocabulary estimation based on observed writings?

Below is a scatter plot of the data set I am dealing with. The X axis is the total number of words per essay for a particular individual, and they Y axis is the number of unique words. In principle, the number of unique words should approach the individuals vocabulary.

I am attempting to find that individual's vocabulary from the data below, but I don't know what kind of a fit would work. A logarithm would have no limit, a quadratic fit doesn't make sense (the gradient should remain non-negative over the entire domain).

In short, I am looking for a decent model to fit the data below, and don't know where to start.

Thank you.

## 1 Answer

In my opinion, this estimation cannot be achieved merely based on this plot, because:

1. From 4000 words onward, the unique words are increasing linearly around 250 per 2K words: (4K, 1.25K), (6K, 1.5K), (8K, 1.75K), (10K, 2K), (12K, 2.25K). So there is not enough evidence to hypothesize an upper-bound for this linear growth,

2. On average, an adult knows 20K-35K unique words, but this plot goes only up to 2K which is far behind the final expected value. The extrapolation from 2K to 20K is unreliable.

Vocabulary of Shakespeare

The estimation of a person's vocabulary is quite complicated. Below is a paper that estimates the vocabulary of Shakespeare. He had used 31K unique words in all of his writtings. The paper estimates that he knew at least 35K more words which he did not use (at least 66K vocabulary). As you see, the estimated vocabulary is only twice the observation, which sheds light on unreliability of going from 2K to 20K and beyond.

1976 Estimating the number of unseen species - How many words did Shakespeare know

• That paper was immensely helpful! And I agree with you, this estimation is not as trivial as I initially thought. The good thing is I have some more data on the way. Thank you very much!
– Mir
Mar 10, 2019 at 10:42