What is valued more in the data science job market, statistical analysis or data processing?

Question

Not sure if this question is OT here. If it is, perhaps move it to meta? If not:

I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning. I've got a doctorate, but not in statistics or computer science. I'm not a software engineer and am only really fluent in R. Maybe I know a bit of C++ and can write a SQL query, but I wouldn't be the guy you'd hire to set up a data pipeline.

There are plenty of engineers who know maybe a thing or two about statistics and machine learning, but who can build apps in Java, manage a spark cluster, set up a distributed database, etc. But they wouldn't be the person you'd hire to do any sort of analysis that isn't already canned in python.

Browsing jobs in the private sector, I see a lot of demand for the second type of data scientist. Is there much demand for the first type? What is the pay differential in the private sector between the two types? Did you start as the first type and become more like the second type "on the job?"

Answer 1

Given your current situation, a data analyst or scientist position makes more sense, but with some practice you can acquire the software skills to become a data- or machine learning engineer. There is a glut of people from academia similar to you, and software companies want people who can actually build software, so it is not surprising you see demand for engineers. But that does not mean you have to become one. Do you want to build software products?

My recommendation would be to make sure you are familiar enough with common applications of statistics in industry (e.g., A/B testing, and time series forecasting) to get a job in analysis, then learn computer science on the side. There is a wide range of salaries, depending on seniority, company and location. You can find specific figures on sites like LinkedIn, Glassdoor, Paysa, Payscale.

Answer 2

Data Science is the new "sexy" job. I got dozens of colleagues who ask me how to become one. Some, come to me and tell me that Andrew Ng's coursera course is hard, and what can they learn instead so they can put it on their CV.

What all this means is that there are a lot of not well prepared people that think they can become a data scientist, and advertise themselves as such, but are really not. Many companies are starting to learn to vet better their candidates because of that.

That said, I am going to answer your question: you have some real skills, you actually understand statistics, which most data scientist I work with don't. Many financial institutions will want people like you, they can find plenty of people who can write Java and set up a cluster and set up a machine learning algorithm without understanding what it does. They are technicians, but do not understand what the data implies. If you can show that you actually can "think" and provide insights, you can be appreciated much more.

In fact, you can command a higher salary. What you need is to learn most used machine learning algorithms, maybe learn some python (easy if you already know R) and tailor your resume for those jobs who require actual thinking. You have real skills.