What to consider before learning a new language for data analysis

Question

I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role in this project.

In view of this I face the following dilemma: Should I learn Python to be able to use its extensive scientific libraries (Pandas, Numpy, Scipy, ...), or should I just dive into similar packages of a language I'm already acquainted with (Racket, or to a lesser extent Scala)?

(Ideally I would learn Python in parallel with using statistical libraries in Racket, but I'm not sure I'll have time for both)

I'm not looking for an answer to this dilemma, but rather for feedback on my different considerations:

My current position is as follows:

In favour of Python:

• Extensively used libraries
• Widely used (may be decisive in case of collaboration with others)
• A lot of online material to start learning it
• Conferences that are specifically dedicated to Scientific Computing with Python
• Learning Python won't be a waste of time anyway

In favour of a language I already know:

• It's a way to deepen my knowledge of one language rather than getting superficial knowledge of one more language (under the motto: you should at least know one language really well)
• It is feasible. Both Racket and Scala have good mathematics and statistics libraries
• I can start right away with learning what I need to know rather than first having to learn the basics

Two concrete questions:

1. What am I forgetting?
2. How big of a nuisance could the Python 2 vs 3 issue be?

Answer 1

Personally going to make a strong argument in favor of Python here. There are a large number of reasons for this, but I'm going to build on some of the points that other people have mentioned here:

1. Picking a single language: It's definitely possible to mix and match languages, picking d3 for your visualization needs, FORTRAN for your fast matrix multiplies, and python for all of your networking and scripting. You can do this down the line, but keeping your stack as simple as possible is a good move, especially early on.
2. Picking something bigger than you: You never want to be pushing up against the barriers of the language you want to use. This is a huge issue when it comes to languages like Julia and FORTRAN, which simply don't offer the full functionality of languages like python or R.
3. Pick Community: The one most difficult thing to find in any language is community. Python is the clear winner here. If you get stuck, you ask something on SO, and someone will answer in a matter of minutes, which is simply not the case for most other languages. If you're learning something in a vacuum you will simply learn much slower.

In terms of the minus points, I might actually push back on them.

Deepening your knowledge of one language is a decent idea, but knowing only one language, without having practice generalizing that knowledge to other languages is a good way to shoot yourself in the foot. I have changed my entire favored development stack three time over as many years, moving from MATLAB to Java to haskell to python. Learning to transfer your knowledge to another language is far more valuable than just knowing one.

As far as feasibility, this is something you're going to see again and again in any programming career. Turing completeness means you could technically do everything with HTML4 and CSS3, but you want to pick the right tool for the job. If you see the ideal tool and decide to leave it by the roadside you're going to find yourself slowed down wishing you had some of the tools you left behind.

A great example of that last point is trying to deploy R code. 'R''s networking capabilities are hugely lacking compared to python, and if you want to deploy a service, or use slightly off-the-beaten path packages, the fact that pip has an order of magnitude more packages than CRAN is a huge help.

Answer 2

From my experience, the points to keep in mind when considering a data analysis platform are:

1. Can it handle the size of the data that I need? If your data sets fit in memory, there's usually no big trouble, although AFAIK Python is somewhat more memory-efficient than R. If you need to handle larger-than-memory data sets, the platform need to handle it conveniently. In this case, SQL would cover for basic statistics, Python + Apache Spark is another option.
2. Does the platform covers all of my analysis needs? The greatest annoyance I've encountered in data mining projects is having to juggle between several tools, because tool A handles web connections well, tool B does the statistics and tool C renders nice pictures. You want your weapon-of-choice to cover as many aspects of your projects as possible. When considering this issue, Python is very comprehensive, but R has a lot of build-in statistical tests ready-to-use, if that's what you need.

Answer 3

According to me, all the factors, you have mentioned are superficial in nature. You have not considered the core of tool selection. In this case, there are 2 aspects, you mentioned:

1. Data analysis - What kind of analysis are you working on? There might be some analysis which are easier in some languages and more difficult in other.

2. Visualization - R provides similar community and learning material (as Python) and has the best visualizations compared to other languages here.

At this stage, you can be flexible with what language to learn, since you are starting from scratch.

Hope this helps.