I have a dataset (600 rows) composed of two columns:

-Summary: which contains the text of a document

-Keywords: which contains the keywords that identify that document.

                                              Summary                     KeyWords_in_Array_wo_insurance
0    court sanction scheme transfer insur reinsur b...                                       [insolvency]
1    immedi custodi sentenc month week impos direct...  [administration of justice, civil evidence, se...
2    motorist injur hit run collis car identifi dri...                         [negligence, road traffic]
3    claimant given permiss continu claim compani a...                      [insolvency, civil procedure]
4    court gave guidanc approach taken applic relea...                           [civil procedure, costs]
5    plaintiff solicitor entitl declar life critic ...                                           [trusts]
6    claimant insur establish requir standard road ...                           [personal injury, torts]
7    minimum indemn requir institut charter account...  [arbitration, civil procedure, costs, accounta...
8    applic secur cost court could take account eve...          [civil procedure, insolvency, cpr, costs]

I want to classify a summary in specific keywords. The keywords are not mutually exclusive.

My code is:

X_train, X_test, y_train, y_test = train_test_split(df_final["Summary"], df_final["KeyWords_in_Array_wo_insurance"], test_size=0.20, random_state=42)

mlb = MultiLabelBinarizer()
y_train_mlb = mlb.fit_transform(y_train)

classifier = Pipeline([
    ('vectorizer', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])

classifier.fit(X_train, y_train_mlb)
y_predicted = classifier.predict(X_test)
all_labels = mlb.inverse_transform(y_predicted)

y_test_mlb = mlb.transform(y_test)

print("Accuracy = ", accuracy_score(y_test_mlb,y_predicted))

I am getting a low accuracy score: 20%

Therefore I am thinking that my classification is not good enough.

The reason it might be that some keywords are used only once. For example, the keywords "animal" or "partnership" or "succession" are used only in 1 row. (Meaning, they are assigned to only one summary)

I think, therefore, that when I split the dataset in training and test, some "lonely keywords" enter the test dataset but not the training one. Thus, the model will never be trained on them.

Is this the reason why my accuracy is so low?

Or am I doing something else wrong?

  • 1
    $\begingroup$ The "lonely words" thing may be part of the problem, but 20% is a little low, so there might also be a problem with the model. You could check on a subset of keywords which occur frequetly (viz. exclude "lonely words") to get a better idea of how your model fits. If the accuracy remains low, you would need to improve your model first. $\endgroup$
    – Peter
    May 20 '19 at 16:46

Accuracy is an awkward measure to use to assess a model predicting classification into multiple classes, and rare events are hard for models to predict well. Outcome categories that exist in your test set but not your training set will of course lead to lower accuracy, but it's far from the only thing that can go wrong.

And, if these outcomes are rare relative to the number of observations you have, it's unlikely that your model is highly predictive but reduced to 20% accuracy based solely on the failure to classify a handful of rare events. It's more likely that the model has fairly poor performance overall. Importantly, this is not necessarily the result of you doing something wrong. Some things are difficult to model, and the best-performing model possible may still have a poor success rate (however you want to measure that).

Without knowing more about your modelling procedure it's hard to make specific suggestions, but a few things come to mind (this is nowhere near an exhaustive list):

Summarizing the documents may not be a very reliable process.

If you have strong protocols for summarizing each, reviewing those might help to find odd edge cases where similar observations get summarized differently in some systematic way, or differing observations being summarized together improperly. If you are manually summarizing each document, it may be worth having multiple people perform the summarization and then evaluation the inter-rater reliability of the summaries.

The documents from which the summaries are derived may differ from one another.

If we're describing highly formulaic documents this might not be an issue. But if the documents are written in prose by different people, it's very possible that issues of personal style and preference on the part of the writers is producing systematic differences from other writers, making it hard to classify a document from an imaginary, generic author.

There could be a mismatch between the number of predictors and outcome classes and the number of observations available to train the model on.

600 observations may be enough for a model classifying a document into one of three categories based on the presence or absence of 15 predictor variables. It is not enough for a model classifying a document into one of 400 categories based on 600 predictor keywords.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.