I have a text classification problem in which i need to classify an answer to a message as either relevant or not.

In the first phase of my calculations, I have already used a SVM to determine if the original message was relevant or not, deciding whether a message contains a hint or question if somebody's twitter account has been hacked.

"Hey @foobar, have you been hacked?"   <-- relevant
"My bank account has just been hacked" <-- not relevant

However, when I want to classify whether the answer is relevant, I would want to have both the original message and the answer as input, right? An answer is relevant in my case if it, in any way, responds to the original message. Is this approach possible using a SVM or any other machine learning tool? I'm using python with the scikit-learn library.

"Hey @foobar, have you been hacked?"
"@barfoo it seems so, thx for suggesting" <-- relevant

"Hey @foobar, have you been hacked?"
"Lose 20 pounds quickly! http://blabla.com" <-- not relevant

I'm not very experienced in this field, so any input would be very appreciated.

  • $\begingroup$ Please, change the topic of your post. $\endgroup$ Apr 24, 2015 at 15:15

1 Answer 1


Both message and answer are your input, so your feature vector should contain information about both.

Here's a simple structure of a possible solution using scikit-learn:

import numpy as np
from sklearn.svm import SVC
from sklearn.feature_extraction import DictVectorizer

dataset = (("Hey @foobar, have you been hacked?",
            "@barfoo it seems so, thx for suggesting",
            True), # True for relevant, False for not relevant
           ("Hey @foobar, have you been hacked?",
            "Lose 20 pounds quickly! http://blabla.com",

def extractMessageFeatures(message):
    # here comes your real feature extraction algorithm
    return { 'message_predicted_spam': False,
             'message_contains_valid_username': True }

def extractAnswerFeatures(answer):
    # here comes your real feature extraction algorithm
    return { 'answer_predicted_spam': False,
             'answer_contains_valid_username': True }

def extractFeatures(data):
    features = []
    for instance in data:
        instanceFeatures = extractMessageFeatures(data[0])
    return features

def trainClassifier(data):
    features = extractFeatures(data)
    vec = DictVectorizer()
    featureVector = vec.fit_transform(features)

    print vec.get_feature_names()
    print featureVector.toarray()

    svc = SVC()
    svc.fit(featureVector, np.array([i[2] for i in data]))
    return svc

clf = trainClassifier(dataset)

# now, you can clf.predict(...)

Now, the hardest part is to decide which features to extract from both messages and answers. It's up to you.

One of the simplest solutions would be to use n-gram features.

Other approach would be to use some spam detection to decide whether answer is spam or not and treat this information as a feature.

You can also use Twitter-specific information (for example, whether users are mentioning each other in their tweets, using the same hashtags etc.).

You can combine these features in whatever fashion you like, of course.

Except of creating feature extraction functionality you need a labeled dataset of messages, answers and relevant/non-relevant labels.

But once you have both (feature extraction functionality and a proper dataset), you're good to go with a task which clearly matches standard machine learning approaches.

  • $\begingroup$ Thank you so much for giving a rough scikit-scheme, that was exactly what I needed! $\endgroup$
    – bmurauer
    Apr 27, 2015 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.