I'm trying to write a script to get the most similar rows in a certain dataframe, based on a single row. Using
The method I need is
kneighbors, which kan return the indices of the closest lying rows in my dataframe if I call it on the
The problem is that my data contains a lot of categorical variables, which
NearestNeighbors can not handle out of the box. In previous projects I always got away with writing a pipeline that did preprocessing and onehotencoding to transform the data to integer/float values. But as not every estimator has a
kneighbors method, a pipeline object does not implement a
What is the best route to both:
- Preprocess the categorical data so it gets onehotencoded and scaled
- Query an unprocessed row to find the
kneighborsof the queryrow.
So that unprocessed row should first get processed the same way the
training data was and after that I want to get the
show the "original" categorical values (so not the processed ones)
cars_data = ... nbrs = NearestNeighbors(n_neighbors=50, algorithm='ball_tree').fit(pd.get_dummies(cars_data)) testpd = pd.DataFrame(columns=['make', 'model', 'body_type', 'transmission', 'fuel','km', 'kw', 'age_months', 'doors','seats', 'color','interior','airconditioning', 'lane_departure', 'led', 'gps', 'panoramic', 'openroof', 'parkassist','soundsystem','standheat','towhook','xenon', 'seatheat','alloywheels']) testpd.loc = ['bmw', '116', 'berline', 'manual', 'benzine', 62450, 80, 10,5,5, 'black', 'fabric', 'YES', 'NO', 'NO', 'NO', 'NO', 'NO', 'YES', 'NO','NO', 'YES', 'NO', 'YES', 'NO'] #should be a variable that can be used to 'query' and get the 50 nearest neighbors closest_neighbors = nbrs.kneighbors(testpd)
This obviously does not work as the testpd is not transformed. I want to do the same transformation as the train data on this testpd, but
OneHotencoding and the scaling is not 'saved' and I'm unable to reapply that on this dataframe with a singular row.