I am working on a multi-label document classification task with a very small data set (180 labeled documents) and a fairly large number of labels (20).

I found that - ignoring label correlations and turning the problem into 20 binary decision problems - tf-idf features fed into a logistic regression work reasonable well.

Since I have domain knowledge I would like to add manual weighs to some features to increase the prediction accuracy. For instance, assume I train a model to recognise the category "Gender", I am thinking about something like this:

'''extract features using tfidf vecorization:'''

vectorizer = TfidfVectorizer(ngram_range = (1,2),min_df = 0.01, max_df = 0.95)
vect = vectorizer.fit(X_train)
X_train = vect.transform(X_train)
X_test = vect.transform(X_test)

'''add additional feature weight'''

weight = 10
position = vect.vocabulary_['woman']
X_train[:, position] *= weight
X_test[:, position] *= weight
position = vect.vocabulary_['gender']
X_train[:, position] *= weight
X_test[:, position] *= weight

The same weighting is then applied to the vectorised unseen data that I want to predict.

This indeed gives me a better accuracy on the test set and also leads to better predictions on the unseen data, however, I feel quite uncomfortable to "manually" change the data.

I have not found any information about such manual feature tuning. Is this something that can be justified?

Bonus question: Tf-idf seems to work the best to build features for longer documents (>2000 characters), however, do you have another suggestion - personal positive experience - to build a feature space for long documents?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.