I have a list of potential job titles e.g. .NET developer, and a corpus of job descriptions that people have written for themselves. I would like to match the job descriptions to the appropriate job title. I have a training set of 650 records. I have tried naive bayes classification and sklearn.MLPclassification with relatively poor results (50% of records classified correctly with 12 possible classes). For both of these methods I used sklearn.TFIDFVectorizer for preprocessing. Is there another method I could try? Perhaps a string similarity matching algorithm or should I try for a simpler method like keyword matching? Here are some examples of a job title and description.

Description: IT Architecture and Strategy Manager & Enterprise Architect

Job title: Solutions Architect

Description: Senior Test Engineer at obfuscated

Job title: Test Analyst

Description: Technical Architecture Delivery Manager at obfuscated

Job title: Service Delivery Manager

  • $\begingroup$ Do you remove rare words with the min_df parameter? How many features do you obtain after TFIDFVectorizer? $\endgroup$
    – Erwan
    Commented Oct 27, 2021 at 17:23

1 Answer 1


Try doing some exploratory data analysis. See if descriptions for different job titles have same keywords. For example I see above 'Architecture' keyword is present for both Solutions Architect & Service Delivery Manager titles. In this case 'Architecture' keyword is not really helping the classifier distinguish between the two titles and hence better off not being included in the model to improve precision. You could try chi-squared test or any other feature selection method to identify words which are really helping you distinguish between the classes and include only those words in your training dataset.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.