2
$\begingroup$

Below is a scatter plot of the data set I am dealing with. The X axis is the total number of words per essay for a particular individual, and they Y axis is the number of unique words. In principle, the number of unique words should approach the individuals vocabulary.

I am attempting to find that individual's vocabulary from the data below, but I don't know what kind of a fit would work. A logarithm would have no limit, a quadratic fit doesn't make sense (the gradient should remain non-negative over the entire domain).

In short, I am looking for a decent model to fit the data below, and don't know where to start.

Thank you.

Scatter plot of data set

$\endgroup$
2
$\begingroup$

In my opinion, this estimation cannot be achieved merely based on this plot, because:

  1. From 4000 words onward, the unique words are increasing linearly around 250 per 2K words: (4K, 1.25K), (6K, 1.5K), (8K, 1.75K), (10K, 2K), (12K, 2.25K). So there is not enough evidence to hypothesize an upper-bound for this linear growth,

  2. On average, an adult knows 20K-35K unique words, but this plot goes only up to 2K which is far behind the final expected value. The extrapolation from 2K to 20K is unreliable.

Vocabulary of Shakespeare

The estimation of a person's vocabulary is quite complicated. Below is a paper that estimates the vocabulary of Shakespeare. He had used 31K unique words in all of his writtings. The paper estimates that he knew at least 35K more words which he did not use (at least 66K vocabulary). As you see, the estimated vocabulary is only twice the observation, which sheds light on unreliability of going from 2K to 20K and beyond.

1976 Estimating the number of unseen species - How many words did Shakespeare know

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ That paper was immensely helpful! And I agree with you, this estimation is not as trivial as I initially thought. The good thing is I have some more data on the way. Thank you very much! $\endgroup$ – Mir Mar 10 '19 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.