I wish to construct feature vectors of words in a document and then calculate their linkage distance to detect anomalies. My question is how can I model these features ? If possible please give an example so I can understand better. Further how distance calculated will vary from the text that is anomalous and the one that is normal ? (Please refer to D. Guthrie, PhD. thesis for more clarification).
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
That is a rather broad question. If you are on a document level, you could (after removing stop words) determine the most important words in your corpus by calculating the tf/idf measure https://en.m.wikipedia.org/wiki/Tf–idf Then you could define a vector over the top n words and represent every document as a vector of the relative frequencies of the top n words in that document and work on these vectors. On a sentence level that could work as well, on a word level obviously not. Anyway, information retrieval and computational linguistics are a rather large research area. Maybe the best way would be to start with a book that provides a good overview.
Since you asked for some more specific recommendations:
If you can get access to (or actually want to buy it), definitely this one: Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition.
You can also take a shot this: Sentiment Analysis and Opinion Mining this is probably not your focus, but it covers feature representation to some extent and is for free.
More like a handbook with practical approaches is: Programming Collective Intelligence, look for it at Google, it is also available for free.
Otherwise if you want to look for yourself for books at a library or so, your keywords are "Computational Linguistics" and "Natural Language Processing".
Hope this helps
-
$\begingroup$ What I usually find helpful for introduction and good references are open stanford courses. For Natural Language Processing, I found this one helpful: (web.stanford.edu/class/cs224n) in the section Textbooks and Readings are futher good references to books if you don't want to go through the slides. $\endgroup$– moligCommented Nov 30, 2016 at 9:35
-
$\begingroup$ Would you please mention a book or two that might help. $\endgroup$ Commented Dec 2, 2016 at 5:28
-