Skills that school doesn't teach you

Question

I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my experience with ML/data science is through Matlab.

Does anybody have insight as to skills that employers will expect me to add before I get a job? I've been looking at postings and it seems like most of the things that I'm not as sharp with are: SQL, Hadoop, Python, and R.

Are Python and R both necessary, or would that be redundant? Is SQL and Hadoop something employers will overlook? Because quite frankly I find learning both to be unbearably boring. Lastly, am I missing anything from this list of skills?

Answer 1

Based on my own experience and in reading what others have written, SQL is one of those skills employers look for, perhaps even assume that you have along with some of the basic skills of communication and teamwork. The main reason is that lots of data are stored in a relational database with SQL being the primary way to extract that data to get it into your models.

Answer 2

I believe technology is cheap and science is expensive. You can learn R, Python, SQL and Hadoop pretty fast (considering that you know programming) but learning statistics, machine learning and the methodology of working with data is difficult and takes time. (which you know based on your background) In my eyes, go and apply for jobs with self-confidence. In the meanwhile, consider learning SQL and Python. They are necessary for jobs in industry.

Answer 3

A large component of data science work in industry is data wrangling. It is quite important to have some basic understanding of data storage systems as you will often have to extract the data you need yourself (unless you work for a large company). Hadoop may not be a necessary skill, but knowing something of relational data stores (SQL) and object storage (No-SQL) will be very useful. A person often needs to be able to process data quickly too, so you will need to know something of optimisations such as indexing.

I work in python and R (as do most practitioners I know personally), but find that python is easier to deploy in a production environment. Much of the work relies on libraries these days, so it useful to know the language's library landscape (when it comes to project timelines, familiarity with your tools reduces the pressure on yourself and your team immensely). It is quite common that people experiment in jupyter/ipython notebooks and make their code available online (e.g. KDNuggets post). We often use notebooks at work and commit them to the code-base to provide the empirical backing for the solution. I would recommend you find some cool notebooks that interface with a database, and see if you can run their code (it is python typically). At least this way you will start to get a feel for more of the grunt work associated with the tools (considering that you already have the theoretical background).

Answer 4

If you find learning SQL and Hadoop unbearably boring, you should not be looking for a data scientist job. Anyhow, feel free to skip Hadoop. There are lots of deployments of Hadoop, but they are being phased out with more modern tech, for example Spark, which is also what most companies use for new deployments. I hope you find learning Spark a bit more bearable. The Edx online courses on ML with Spark are actually very nice.

Answer 5

Hadoop and SQL are things that you will pick up reasonably quickly and in my experience are far from necessary for a lot of jobs.

I would focus on Python or on R, not on both. A lot of employers are Python and R and allow you to choose yourself as long as it's on one or both. I feel like Python is growing faster in the Data Science community than R and is in my opinion a much more sophisticated language and has a wider support for things that help with data science but are not directly related.

As jab already mentioned, I think soft skills are very important, although difficult to learn without real working experience.

The best thing to prepare in my opinion is just by doing some project(s) and figuring things out along the way. Join a Kaggle competition or find some personal project. I spend hours and hours working on those and I keep coming across problems that I haven't faced before, requiring me to read papers, implementing some new ideas, trying things out. This also allows you to build up a github portfolio to show off some cool projects you have been working on.

Answer 6

SQL / Big Data Tools

Getting and cleaning data is a central part of a data scientist's job. My team (data science at a mid-to-large internet company) regularly deals with data wrangling at scale -- combining terabytes of browsing history with a multitude of other data sources, structured and unstructured, to investigate problems. SQL and Spark/Hive/Hadoop are nearly daily parts of our workflow. We filter new candidates heavily on their ability to use SQL intelligently and favor those who have worked with big data technologies in production environments.

On a side note, distributed processing/storage is a fascinating research area and the mechanics behind databases are pretty cool. Perhaps you can make it a bit more interesting for yourself by not just focusing on the semantics of SQL, but also some under-the-hood bits, like multi-pass algorithms for joins and so forth. You could read the Amazon DynamoDB paper or Google's BigTable paper for an intro to big, distributed databases.

Programming environment

Our team is python heavy, though we are using Spark more and more. It's got a great ecosystem for working with big data and has great machine-learning support via scikit-learn and the like. We often prototype in jupyter notebooks and scale solutions by building python apis that run our jobs or handle incoming requests. Other data science teams here use R. Pick one, do a few projects in it, and do a few projects in the other. You'll figure out which syntax and style works best for you.

Try to train yourself not to code like a grad student anymore, either. :)

Answer 7

Python is more suited for a 'real' data scientist work whose work needs to be productized and used in a software or website. We need to remember that ultimately whatever we do needs to be productized. Although R has Shiny library that can be used for web hosting, we need to remember that a website will not have only machine learning component, it will have many other aspects, which cannot be built in R, but needs a full fledged software development language. I sent my resume to senior level data scientist for job application and the response i received is in the linked question. Why do internet companies prefer Java/Python for data scientist job?

Answer 8

AFTER graduation, you will [continue to] learn how to learn, how to accelerate your learning process with smart people who get smarter because they accelerate how they learn ... away from academia, without institutional support, in difficult environments ... each of us must transcend our own algorithmic or predictable approach to learning or how our machine has been programmed in the past to recognize patterns and learn.

How you collaborate, communicate, work with teams of solid, qualified, antifragile LEARNERS will shape your career, success, satisfaction, friendships, life ... and our species.

Smart people are often suckers for the propaganda and comforts of respectable human systems, such as academia or bureaucratic machines built on piles of smart people who settle for calcifying into smart, predictable cogs ... but the world is chaotic and throws sand into the working of inflexible systems and smart cogs -- so how you learn will determine whether chaos serves you or defeats you. Learn to think in a generally antifragile manner rather than simply being resilient or defensively robust ... as you wrangle data, remember that the most interesting and useful things will tend to happen in Extremeistan.

Assuring quality of the data you are wrangling [or that some intern has wrangled] for the analysis is always going to be essential; the GIGO rule will always take precedence over the sophisticated wonkery. That is why it is so necessary for us to step back, see our roles in a larger picture. Being proficient in skill like Python or experimental design is awesome and necessary -- but do not settle for being pigeon-holed as just an expert in _____ or a phenomenal code jockey.

Data, like humans, have more interesting stories to tell than just the narrow questions we want to ask AND just because we have a bias, it does not mean that it's necessary to wrangle up compliant data to confirm it. Outliers need to be listened to; there's a reason why they show up, why they persist. Listen to extreme, challenging, heretical points of view ... especially when the local inquisition is angry about the uncomfortable someone can prove that the planets do not orbit around them. Not all uncomfortable, unpopular opinions should taken seriously, but some are absolutely essential and necessary truth. Beware of the zealots, even brilliant cartoonists, who claim to think independently when 97% of them conform to a generally-accepted truth and adamantly demand that others fall into line ... sometimes, cartoons are just metaphors for how seriously we should take cartoons.