I have a large amount of CSV files, an example of which (for job titles) is listed below. The data is noisy (there are misspellings, difference in capitalisation, missing values, and they are not well-formed (some files have headers, some have not, and if headers are present, they don't always agree on name). I have gazetteers available.

...,IT Manager,...
...,Senior IT Manager,...
...,it manager,...
...,IT managre,...
...,junior IT managre,...
...,Business Consultant,...
...,Business consultent,...

I have a finite set of entity types (First Name, Last Name, Location etc.), and the task is: Given the content of this column, which entity type, if any, does it represent? (It is the first step in a processing pipeline.)

It is a form of named entity recognition [supervised learning classification] task, but all the papers I have read about named entity recognition uses conditional random fields or maximum entropy for natural language tasks, and I don't think this count as natural language.

What would be the appropriate approach/ML algorithm for such a task? How should my training data be formatted?


1 Answer 1


Yes, it might not be exactly natural Language Understanding but CRF is an excellent algorithm to train Named Entity Recognition tasks and is the stamdard model used by Stanford NLP group. You can try out their NER tagger here. If you want something that accounts for language understanding then there are certain papers who have trained recurrent neural network architectures such as LSTM and Bi-directional RNNs. Look at this paper. I must tell you that Named Entity Recognition is an incredibly hard problem and if you want to use Deep Learning architectures, it will require tremendous amount of data. I will suggest try out the Stanford NER tagger, since your data doesn't have much of a sequential nature, I am certain it will perform well. In the end here is part 1 of an excellent blog post that goes in detail about training your own NER model using the Stanford NER tool.

  • $\begingroup$ For such tasks, what should be considered as the features? Would they be 'IT Manager,Senior IT Manager,...'? Also, would Naive Bayes classifier be a prudent choice? If there features are what I mentioned, there should be no independence between them, correct? $\endgroup$
    – normannen
    Commented Feb 8, 2017 at 10:01
  • $\begingroup$ By features do you mean the entity (class) ? $\endgroup$ Commented Feb 8, 2017 at 10:21
  • $\begingroup$ I mean the input features on which the model will be trained. My understanding is it would be all the values of each column along with an associated label. So "Business Manager/JOBTITLE,it manager/JOBTITLE" etc. But how would that work if I try to predict misspelled titles not present in the training data? I agree with the fact that my data doesn't have a sequential nature, but the words in the job titles themselves are still dependent, no? $\endgroup$
    – normannen
    Commented Feb 8, 2017 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.