I was wondering whether it's possible to compute the some sort of pointwise mutual information between lemma and its surface form.

First if we assume,

p('to go') = count('to go') / sum(all lemmas)

p('went') = count('went') / sum(all words)

Breakpoint here, since every word comes with its respective lemma, we have the condition that

sum(all lemmas) == sum(all words)

The joint probability is also a little hard to normalize

# count of "went" being lemmatize to "to go
p('went', 'to go') = count('went'->'to go') / sum(all words)

The catch here is since "went" always lemmatize to "to go", we have special condition of

count('went'->'to go')  == count('went')

The reverse is not true though since the "to go" can be realized as different surface forms.

In that case, it's really awkward, if we put it all together

 p('went', 'to go') = p('went') = count('went') / sum(all words)
 p('to go') = count('to go') / sum(all words)

 PMI('went', 'to go') 
 =  p('went', 'to go') / (p('to go') * p('went') 
 =  p('went') / (p('to go') * p('went') 
 =  1 / p('to go') 

But that would assign the same information value for all surface forms given the same lemma.

I think I've made some mistakes in how I'm accounting for the joint or independent probabilities. Could anyone advise how to approach an information score for lemma and its surface?

Or is the information score between lemma and its surface uninformative thus the conclusion of inverse probability of the lemma?

One possible solution is to change the normalization value for the joint probability where:

p('went', 'to go') = count('went') / count('to go')

Then the PMI equation would be:

PMI('went', 'to go') 
=  p('went', 'to go') / (p('to go') * p('went') 
=  (p('went') / (p('to go')) / (p('to go') * p('went') 
=  1 / p('to go')**2

Still the final mutual information disregard the surface probability =(

  • $\begingroup$ Possibly, I'm abusing the p(A,B) and making it p(A|B) when assuming count('went'->'to go') == count('went')? $\endgroup$ – alvas Aug 8 '18 at 7:05
  • $\begingroup$ Or maybe the normalization value for the joint probability should be lemma, i.e. p('went', 'to go') = count('went') / count('to go')? $\endgroup$ – alvas Aug 8 '18 at 7:10
  • $\begingroup$ do you score the probablity that you parsed went? $\endgroup$ – amirouche Aug 8 '18 at 9:53
  • $\begingroup$ @amirouche I might be misunderstanding you but this is a count based probability and doesn't have any parsing involved. $\endgroup$ – alvas Aug 8 '18 at 14:23

The result that you are calling awkward is indeed what we expect from the definition of PMI. Here are some identities involving PMI:

$\operatorname{pmi}(x;y) \equiv \log\frac{p(x,y)}{p(x)p(y)} = \log\frac{p(x|y)}{p(x)} = \log\frac{p(y|x)}{p(y)}$

Let $x=$'went' and $y=$'to go' and use the last expression. $p("to go"|"went") = 1$, and voilà, only $p("to go")$ remains in the denominator of the expression.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.