I have a basic doubt in construction of co-occurence matrix from text. Coocurence matrix X is defined as :

Xij : number of times word i occurs in context of word j in a window of size w.

Now is this context limited to sentence i.e do we context word of a main word in a sentence only or across the whole document?

For ex:

Consider the corpus :

human interface computer
survey user computer system response time
eps user interface system
system human system eps
user response time
graph trees
graph minors trees
graph minors survey
I like graph and stuff
I like trees and stuff
Sometimes I build a graph
Sometimes I build trees

Now for word graph and context window 10 do we look for only 4th last line or do we search for words across line also considering whole doc as one line?


Whether to window only over sentences or over the whole corpus depends on the context of your problem and on the structure of your corpus. In this corpus the order of your sentences does not matter, if they were switched around it would still be the same corpus. This means that you should stick to only the lines and not see it as one line. If your corpus would be a book for example, spilling over sentences makes a lot more sense. If you are unsure, try both, it is just a hyperparameter of your model.

  • $\begingroup$ Yeah, makes sense. Thanks, that answers my question. I'll try both approaches. $\endgroup$ – SHASHANK GUPTA Feb 28 '16 at 11:54
  • $\begingroup$ I don't have a lot of experience with this but I could imagine you could even do some kind of hybrid where you weigh the words depending on if they appear in the same sentence or in another sentence still within the window $\endgroup$ – Jan van der Vegt Feb 28 '16 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.