I'm trying to understand the lambda collocation scoring metric from the quanteda package in R, but the explanation in the documentation is incredibly difficult to understand.

Can anyone explain it better?

Having an example worked out on a toy set of text would help too, like something with 5-10 words. This line (and the one right before it) is especially confusing:

The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci.


I will try to illustrate the lambda collocation metric, first we have to define a function c(x, z) that receives two K-word expressions where x is the K-word expression you want to score and z corresponds to all possible values of a K-word expression in the corpus. c(x, z) returns a binary vector (j1, ..., jk) such that ji = 1 if xi == zi, 0 otherwise. Now a binary vector can represent a decimal number up to 2^K = M, so c(x, z) = ci means that the binary vector returned was equal to the binary representation of the decimal number i. Now as it says on the documentation ni corresponds to the number of times the c(x, z) function was equal to ci through the corpus. Now an example, consider the following sentence:

this sentence is an example for this sentence

The target expression is "this sentence" so K=2. You have the following possible values for z:

this sentence
sentence is
an example
example for
for this
this in
in this
this sentence

Given that K=2, the possible values for c(x, z) are 00 = 0, 01 = 1, 10 = 2 and 11 = 3. Then considering the target expression and all the values of z above you have the following, (for x = "this sentence"):

c(x, this sentence) = 11 = c3
c(x, sentence is) = 00 = c0
c(x, an example) = 00 = c0
c(x, example for) = 00 = c0
c(x, for this) = 00 = c0
c(x, this in) = 10 = c2
c(x, in this) = 00 = c0
c(x, this sentence) = 11 = c3

So we have 5 times c0 which means n0 = 5, 2 times c3 means n3 = 2 and n2 = 1. Finally bi is the number of bits equals to 1 in c1 which b0 = 0, b1 = 1, b2 = 1, b3 = 2. This completes all the values you need for computing the lambda collocation scoring metric. Also take into account you could add smoothing to ni to avoid 0 counts. Now with respect to

The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci.

The definition of a contingency table for the array data is simply the number of times a given value appears (i.e. its frequency). See this and the usage of the table command for more information. For the example above we have the following array data:

c3 c0 c0 c0 c0 c2 c0 c3

So the corresponding contingency table is:

c0 | c1 | c2 | c3
 5 |  0 |  1 |  2 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.