Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend to reach out for C/C++/low-level languages. The possibility of tailoring the code to the needs seems to be what makes these languages so appealing for BigData -- be it concerning memory management, parallelism, disk access, or even low-level optimizations (via assembly constructs at C/C++ level).
Of course such set of benefits would not come without a cost: writing the code, and sometimes even reinventing the wheel, can be quite expensive/tiresome. Although there are lots of libraries available, people are inclined to write the code by themselves whenever they need to grant performance. What disables performance assertions from using libraries while processing large databases?
For example, consider an entreprise that continuously crawls webpages and parses the data collected. For each sliding-window, different data mining algorithms are run upon the data extracted. Why would the developers ditch off using available libraries/frameworks (be it for crawling, text processing, and data mining)? Using stuff already implemented would not only ease the burden of coding the whole process, but also would save a lot of time.
In a single shot:
- what makes writing the code by oneself a guarantee of performance?
- why is it risky to rely on a frameworks/libraries when you must assure high performance?