3
$\begingroup$

I'm new to all this and am putting together a learning project. I've decided on finding similarities between users in a data set such as http://en.wikipedia.org/wiki/Enron_Corpus. After doing a bit of research, I also came across Dataset for Named Entity Recognition on Informal Text. So I'm not short of data or a goal, I need to understand high-level techniques to get there.

One valuable comment noted that this question appears too broad. What I was hoping to find with this question was the breadth of techniques I should focus research on, not answers that are immediately implementable. Please consider vague answers as entirely appropriate!!

Expanding on the goal, I am hoping to discover which authors might have affinity toward each other, or conversely do not care much for each other. So I will definitely need to start with Named Entity Recognition and build a means to organize the documents against those entities. Beyond that, I am not so sure.

What high level concepts should I be looking at? Thanks!

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ I think that your question is too broad to expect comprehensive enough answers. I would recommend to research yourself major high-level ML techniques (Wikipedia set of relevant articles is a decent starting point) and then formulate more narrow question(s). Since you've mentioned NER, you might find helpful my related answers here and (linked within) here. $\endgroup$
    – Aleksandr Blekh
    Commented May 10, 2015 at 6:35
  • 2
    $\begingroup$ Thanks @AleksandrBlekh, I should have been more specific that I am looking for high level direction than low level implementation. Your comment helps a lot with that. I use Wikipedia a lot, but it never crossed my mind to look there on this subject. I'll update the question to better reflect what I am after and follow with new questions in the future. $\endgroup$
    – Brian Topping
    Commented May 10, 2015 at 6:53
  • $\begingroup$ You are welcome. Always glad to help. $\endgroup$
    – Aleksandr Blekh
    Commented May 10, 2015 at 6:55
  • $\begingroup$ Interesting project. The free Udacity class - Intro to Machine Learning (udacity.com/course/intro-to-machine-learning--ud120) also explores Enron Corpus. To discover which authors might have affinity, maybe one can start with examining the number of emails between two people, assuming the number of emails would indicate that they cared about each other enough so they didn't avoid emailing each other as much as possible. What's your learning goal in this project? $\endgroup$
    – hostjc
    Commented May 29, 2015 at 20:50
  • 1
    $\begingroup$ Suppose you find that some people exchanged emails a lot (say 100 email exchanges or above). Can these people be further divided into two groups according to certain features, such as the average time to reply an email (within three days or longer than that)? In addition to clustering, you can do supervised learning - search newspaper stories to find some people who cared about each other and other people who didn't care that much, then check the characteristics of emails from these two groups. $\endgroup$
    – hostjc
    Commented May 29, 2015 at 21:30

1 Answer 1

Reset to default
1
$\begingroup$

As you accept vague answers: Sranford NLP tools are strong for this kind of stuff. NER, POS Tagger maybe, Parsers, etc. Now for machine learning itself, I would try looking up at WEKA, it's got a lot of filtering, classifiers and clustering methods, including StringToWordVector filters, that are, in my opinion, fundamental to text classification. Mostly, the tags you should be looking for are Text Categorization, Natural Language Processing and even Sentiment Analysis if you will.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Sorry: ***STANFORD NLP $\endgroup$
    – Henrique Nader
    Commented Jun 5, 2015 at 5:20
  • $\begingroup$ Hmm, thanks! I've been working with Epic, a tool written in Scala from scalanlp.org.... $\endgroup$
    – Brian Topping
    Commented Jun 6, 2015 at 9:06
  • $\begingroup$ What language do you use? $\endgroup$
    – Henrique Nader
    Commented Jun 8, 2015 at 19:44
  • $\begingroup$ I'm using Scala, but I didn't want to make that a predicate on the high level technique that would be good to pursue on a project like this. $\endgroup$
    – Brian Topping
    Commented Jun 9, 2015 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.