I've got a Excel file having columns Name, Surname and Community, having around 31 thousand entries. For my college project, the task I have is to write a program to predict "Community" of a person, given their name and surname. For example, the first entry in the file is the name: Yogita Singh. If someone enters a name like Yogta Singha, then approximately it should predict that that person belongs to the Punjabi communtiy.

I did manage to write a program, which does the prediction using the Levenshtein distance. I gave more weightage to the surname (10 times more), than the first name, because surname and community has a greater correlation than first-name and community (naturally, because some first-names are very common and used by people from many different communities).

Any ideas how to improve the accuracy of my model? Has there been any relevant theoretical study regarding name - ethnicity mapping/correlation models? I think Levenshtein distance is a good start but I'm not sure the exact weights I should give to the surname and first name. Also, perhaps I need to give more weight to the characters towards the beginning of the surname, rather than characters towards the end.

Any suggestions regarding how to improve the accuracy of my model will be greatly appreciated. Thanks.


Feature engineering

The question you want to ask yourself is "When you look at a name how do you as a human deduce the community?" I would claim that this is possible by looking at the way the letters are grouped together. For example "aoui" and "wi" is very common in India and Arabic, whereas "Zh" would be very common in Chinese. So how can we extract these types of features?


n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

This would capture the distribution of letter combinations which are present for each community.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy