I wish to construct feature vectors of words in a document and then calculate their linkage distance to detect anomalies. My question is how can I model these features ? If possible please give an example so I can understand better. Further how distance calculated will vary from the text that is anomalous and the one that is normal ? (Please refer to D. Guthrie, PhD. thesis for more clarification).


That is a rather broad question. If you are on a document level, you could (after removing stop words) determine the most important words in your corpus by calculating the tf/idf measure https://en.m.wikipedia.org/wiki/Tf–idf Then you could define a vector over the top n words and represent every document as a vector of the relative frequencies of the top n words in that document and work on these vectors. On a sentence level that could work as well, on a word level obviously not. Anyway, information retrieval and computational linguistics are a rather large research area. Maybe the best way would be to start with a book that provides a good overview.

Since you asked for some more specific recommendations:

If you can get access to (or actually want to buy it), definitely this one: Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition.

You can also take a shot this: Sentiment Analysis and Opinion Mining this is probably not your focus, but it covers feature representation to some extent and is for free.

More like a handbook with practical approaches is: Programming Collective Intelligence, look for it at Google, it is also available for free.

Otherwise if you want to look for yourself for books at a library or so, your keywords are "Computational Linguistics" and "Natural Language Processing".

Hope this helps

  • $\begingroup$ What I usually find helpful for introduction and good references are open stanford courses. For Natural Language Processing, I found this one helpful: (web.stanford.edu/class/cs224n) in the section Textbooks and Readings are futher good references to books if you don't want to go through the slides. $\endgroup$
    – molig
    Nov 30 '16 at 9:35
  • $\begingroup$ Would you please mention a book or two that might help. $\endgroup$ Dec 2 '16 at 5:28
  • $\begingroup$ I edited my answer. $\endgroup$
    – molig
    Dec 2 '16 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.