I am working on a dataset, and i would like to create a model that would predict salary.

Columns are as follows:

Index(['ID', 'Salary', 'DOJ', 'DOL', 'Designation', 'JobCity', 'Gender', 'DOB',
       '10percentage', '10board', '12percentage', '12board', 'CollegeTier',
       'Degree', 'Specialization', 'collegeGPA', 'CollegeCityTier',
       'CollegeState', 'English', 'Logical', 'Quant', 'Domain',
       'ComputerProgramming', 'ElectronicsAndSemicon', 'ComputerScience',
       'MechanicalEngg', 'ElectricalEngg', 'TelecomEngg', 'CivilEngg',
       'conscientiousness', 'agreeableness', 'extraversion', 'nueroticism',


The train dataset contains around 3000 entries. For the feature 'JobCity', there are around 200 unique values. Out of 3000 entries, 180 are cities whose value counts are below 10. For example, the city 'Agra' occurs only twice and 'Ranchi' occurs only once. I plan to encode all the cities to feed into the machine learning (polynomial regression) model. How should the cities with value counts less than 10 be handled?


There are two ways to handle this:

  1. Aggregating values and feature engineering

Instead of simply inputting the individual city, you could try to aggregate this value into a new variable "JobCitySize" to cluster cities by inhabitant size or a variable "Region" to cluster by geographic region. This will increase cases and also help solve the question of how to deal with unseen data.

  1. Defining NA or unseen values

With any model and categorical encoding you have to think ahead what you want to do with "unseen data" in your case cities that aren't present in your training data set. Do you want to set them as NA, give a standard value ("Other"), etc.? Whatever process you use for these values you could also use for any value in your data set that falls below a threshold of "usefulness".

  • $\begingroup$ Regarding the first suggestion i.e clustering according to region, do you suggest that i should manually go through every city (those 180 entries), google what region they lie in, and then write code to cluster them? Regarding the second suggestion, is it okay to give a standard value "Other" to all the 180(out of 3000) entries? Will that affect the model's accuracy too much? $\endgroup$ – Amethyst Jul 14 '20 at 13:18
  • $\begingroup$ What way to cluster is best is up to your data and resources. There are libraries that are able to match city names to GPS locations which could then be used to group by proximity (e.g. geopy) and you could crawl Wikipedia to get inhabitant size. Grouping all of these cities into "Other" seems fine to me but could lead to problems of overfitting (e.g. in your data set only small cities are in "Other" but in unseen data a big city gets in there), the only way to know is to try out the model and look at CV scores. Whether or not heavy feature engineering is worth it depends on your case. $\endgroup$ – Fnguyen Jul 14 '20 at 13:23
  • $\begingroup$ Ah yes i get your point regarding grouping all cities under 'Others'.. Thank you! I'll look into the libraries :) $\endgroup$ – Amethyst Jul 14 '20 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.