I'm new to data science field and interested in performing prediction using clickstream data. In Practical Lessons from Predicting Clicks on Ads at Facebook paper section 3.1, a method called Decision Tree Feature Transforms confused me a lot. After searching around, there is no website / book / paper describing about this method. Can anyone give me link references describing this method in detail?


What the article mentions is random tree embeddings.

This is a kind of unsupervised feature extraction method based on random trees.

In python you can find the Scikit-learn implementation.

From the documentation:

Transform your features into a higher dimensional, sparse space. Then train a linear model on these features.

First fit an ensemble of trees (totally random trees, a random forest, or gradient boosted trees) on the training set. Then each leaf of each tree in the ensemble is assigned a fixed arbitrary feature index in a new feature space.

These leaf indices are then encoded in a one-hot fashion. Each sample goes through the decisions of each tree of the ensemble and ends up in one leaf per tree. The sample is encoded by setting feature values for these leaves to 1 and the other feature values to 0. The resulting transformer has then learned a supervised, sparse, high-dimensional categorical embedding of the data.

You may also find these references very helpful:

  1. https://gdmarmerola.github.io/forest-embeddings/

  2. https://blog.davidvassallo.me/2019/08/06/3-uses-for-random-decision-trees-forests-you-maybe-didnt-know-about/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.