I have a dataframe like as shown below
ID,Name,year,output 1,Test Level,2021,1 2,Test Lvele,2022,1 2,dummy Inc,2022,1 2,dummy Pvt Inc,2022,1 3,dasho Ltd,2022,1 4,dasho PVT Ltd,2021,0 5,delphi Ltd,2021,1 6,delphi pvt ltd,2021,1 df = pd.read_clipboard(sep=',')
My objective is
a) To replace near duplicate strings using a common string.
For example - let's pick couple of strings from
Name column. We have
dummy Inc and
dummy Pvt Inc. These both have to be replaced as
I manually prepared a mapping df
map_df like as below (but can't do this for big data)
Name,correct_name Test Level,Test Test Lvele,Test dummy Inc,dummy dummy Pvt Inc,dummy dasho Ltd,dasho dasho PVT Ltd,dasho delphi Ltd,delphi delphi pvt ltd,delphi
So, I tried the below
map_df = map_df.set_index(Name) df['Name'] = df['Name'].map(map_df) # but this doesn't work and throws error
Is creating mapping table the only way or is there any NLP based approach?
I expect my output to be like as below
ID,Name,year,output 1,Test,2021,1 2,Test,2022,1 2,dummy,2022,1 2,dummy,2022,1 3,dasho,2022,1 4,dasho,2021,0 5,delphi,2021,1 6,delphi,2021,1