Since these libraries can turn CPU arrays into GPU tensors, could you parallelize (and therefore accelerate) the calculations for a decision tree? I am considering making a decision tree class written in Tensorflow/Pytorch for a school project, but I want to be certain that it makes sense.
Not directly, mostly because the structure of a decision tree doesn't lend itself to GPU parallelisation, and is better suited to CPUs. Even the most established decision tree algorithms use CPUs.
- GPUs can only perform a small subset of operations at high performance, the most important being matrix multiplication, which is the fundamental building block of a neural network.
- Decision trees don't train by gradient descent, but an iterative process of partitioning the dataset. As such, they need access to the entire dataset in memory (rather than batches).
- Decision trees are not easily differentiable without amendment.
The Deep Neural Decision Forest paper does address these issues in a simple and clever way:
This is interesting as it does allow trees to be learnt with gradient descent, but it's unlikely to be faster than existing CPU bound tree algorithms.