I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to perform a classification algorithm taking all the inputs to determine the income range. I need to build a dictionary of variables to mapped variables and use a map function to map the variables to numbers for processing. Essentially, I would my dataset to be in a numerical format so that I can work on implementing the models.
In the data set, there are categorical columns like education, marital status, working class etc. Can someone tell me how to convert them into numerical columns in pyspark?
workclass = {'?':0,'Federal-gov':1,'Local-gov':2,'Never- worked':3,'Private':4,'Self-emp-inc':5,'Self-emp-not-inc':6,'State-gov':7,'Without-pay':8}
I created a sample dictionary with key value pairs for work class. But, I don't know how to use this in a map function and replace the categorical data in the CSV file with the corresponding value.
wc = pd.read_csv('PATH', usecols = ['Workclass'])
df = pd.DataFrame(wc)
wcdict = {' ?':0,' Federal-gov':1,' Local-gov':2,' Never-worked':3,' Private':4,' Self-emp-inc':5,' Self-emp-n-inc':6,' State-gov':7,' Without-pay':8}
df_new = df.applymap(lambda s: wcdict.get(s) if s in wcdict else s)
print(df_new)
This is the code I have written in normal python to convert the categorical data into numerical data. It works fine. I want to do the conversion in spark context. And, there are 9 categorical columns in the data source. Is there a way to automate the dictionary update process to have a KV pair for all 9 columns?