I'm trying to write a script to get the most similar rows in a certain dataframe, based on a single row. Using scikit-learn

The method I need is kneighbors, which kan return the indices of the closest lying rows in my dataframe if I call it on the NearestNeigbors algorithm.

The problem is that my data contains a lot of categorical variables, which NearestNeighbors can not handle out of the box. In previous projects I always got away with writing a pipeline that did preprocessing and onehotencoding to transform the data to integer/float values. But as not every estimator has a kneighbors method, a pipeline object does not implement a kneighbors method.

What is the best route to both:

  • Preprocess the categorical data so it gets onehotencoded and scaled
  • Query an unprocessed row to find the kneighbors of the queryrow.

So that unprocessed row should first get processed the same way the training data was and after that I want to get the kneighbors which show the "original" categorical values (so not the processed ones)

cars_data = ...

nbrs = NearestNeighbors(n_neighbors=50, algorithm='ball_tree').fit(pd.get_dummies(cars_data))

testpd = pd.DataFrame(columns=['make', 'model', 'body_type', 'transmission', 'fuel','km', 'kw', 'age_months', 'doors','seats', 'color','interior','airconditioning', 'lane_departure', 'led', 'gps', 'panoramic', 'openroof', 'parkassist','soundsystem','standheat','towhook','xenon', 'seatheat','alloywheels'])
testpd.loc[0] = ['bmw',       '116',        'berline',     'manual',   'benzine',   62450,  80, 10,5,5, 'black', 'fabric', 'YES', 'NO', 'NO', 'NO', 'NO', 'NO', 'YES', 'NO','NO', 'YES', 'NO', 'YES', 'NO'] #should be a variable that can be used to 'query' and get the 50 nearest neighbors

closest_neighbors = nbrs.kneighbors(testpd)

This obviously does not work as the testpd is not transformed. I want to do the same transformation as the train data on this testpd, but OneHotencoding and the scaling is not 'saved' and I'm unable to reapply that on this dataframe with a singular row.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.