A colleague and I have conducted some preliminary studies on the performance differences between pandas and data.table. You can find the study (which was split into two parts) on our Blog (You can find part two here).
We figured that there are some tasks where pandas clearly outperforms data.table, but also cases in which data.table is much faster. You can check it out yourself and let us know what you think of the results.
If you don't want to read the blogs in detail, here is a short summary of our setup and our findings:
data.table on 12 different simulated data sets on the following operations (so far), which we called scenarios.
- Data retrieval with a select-like operation
- Data filtering with a conditional select operation
- Data sort operations
- Data aggregation operations
The computations were performed on a machine with an Intel i7 2.2GHz with 4 physical cores, 16GB RAM and a SSD hard drive. Software Versions were OS X 10.13.3, Python 3.6.4 and R 3.4.2. The respective library versions used were 0.22 for pandas and 1.10.4-3 for data.table
Results in a nutshell
data.tableseems to be faster when selecting columns (
pandason average takes 50% more time)
pandas is faster at filtering rows (roughly 50% on average)
data.table seems to be considerably faster at sorting (
pandas was sometimes 100 times slower)
- adding a new column appears faster with
- aggregating results are completely mixed
Please note that I tried to simplify the results as much as possible to not bore you to death. For a more complete visualization read the studies. If you cannot access our webpage, please send me a message and I will forward you our content. You can find the code for the complete study on GitHub. If you have ideas how to improve our study, please shoot us an e-mail. You can find our contacts on GitHub.