43
$\begingroup$

I would consider myself a journeyman data scientist. Like most (I think), I made my first charts and did my first aggregations in high school and college, using Excel. As I went through college, grad school and ~7 years of work experience, I quickly picked up what I consider to be more advanced tools, like SQL, R, Python, Hadoop, LaTeX, etc.

We are interviewing for a data scientist position and one candidate advertises himself as a "senior data scientist" (a very buzzy term these days) with 15+ years experience. When asked what his preferred toolset was, he responded that it was Excel.

I took this as evidence that he was not as experienced as his resume would claim, but wasn't sure. After all, just because it's not my preferred tool, doesn't mean it's not other people's. Do experienced data scientists use Excel? Can you assume a lack of experience from someone who does primarily use Excel?

Improve this question
$\endgroup$
8
  • $\begingroup$ Most data science job ads ask for specific skills, like R, Hadoop, whatever. Did you neglect to mention this in your ad? Unless your new Data Scientist is going to work in a bubble then he or she will have to work with the team, and probably need to work with the standard team software... $\endgroup$
    – Spacedman
    Apr 4, 2015 at 8:50
  • 1
    $\begingroup$ well if they won't use \LaTeX{} then i wouldn't hire 'em. just kidding... $\endgroup$
    – aeroNotAuto
    Apr 4, 2015 at 17:47
  • 1
    $\begingroup$ @Spacedman: I provided the story for anecdotal context but am really more interested in people's views on excel than I am hiring tips. Our team is free to use whatever tools we like. $\endgroup$
    – JHowIX
    Apr 5, 2015 at 23:15
  • 1
    $\begingroup$ Yes, see here. For the joke impaired, see here too. $\endgroup$
    – Dirk Eddelbuettel
    Apr 11, 2015 at 14:12
  • 1
    $\begingroup$ Regardless of years specified, I'd expect a Pro/Con list of at least three tools from a data scientist. They need to show capacity to investigate, weight options, and communicate resolutions. Even, or especially, in an interview, I'd expect to see real engagement and a capacity to expand past a potentially great, but currently lacking, interview question. $\endgroup$
    – Dave
    Feb 23, 2016 at 23:07

10 Answers 10

Reset to default
35
$\begingroup$

Most non-technical people often use Excel as a database replacement. I think that's wrong but tolerable. However, someone who is supposedly experienced in data analysis simply can not use Excel as his main tool (excluding the obvious task of looking at the data for the first time). That's because Excel was never intended for that kind of analysis and as a consequence of this, it is incredibly easy to make mistakes in Excel (that's not to say that it is not incredibly easy to make another type of mistakes when using other tools, but Excel aggravates the situation even more.)

To summarize what Excel doesn't have and is a must for any analysis:

  1. Reproducibility. A data analysis needs to be reproducible.
  2. Version control. Good for collaboration and also good for reproducibility. Instead of using xls, use csv (still very complex and has lots of edge cases, but csv parsers are fairly good nowadays.)
  3. Testing. If you don't have tests, your code is broken. If your code is broken, your analysis is worse than useless.
  4. Maintainability.
  5. Accuracy. Numerical accuracy, accurate date parsing, among others are really lacking in Excel.

More resources:

European Spreadsheet Risks Interest Group - Horror Stories

You shouldn’t use a spreadsheet for important work (I mean it)

Microsoft's Excel Might Be The Most Dangerous Software On The Planet

Destroy Your Data Using Excel With This One Weird Trick!

Excel spreadsheets are hard to get right

Improve this answer
$\endgroup$
9
  • $\begingroup$ For looking at data and quickly analyzing it, are there tools widely accepted by pros as comparable but better than Excel? I'm a newbie data scientist, and I've mostly been using (Postgre)SQL, but something like Excel can be faster to work with if you're just trying things out. $\endgroup$
    – sudo
    Jan 20, 2017 at 17:50
  • 1
    $\begingroup$ Also, I have to complain that CSV is not a standard. You really have to make sure that whatever is opening it agrees with whatever produced it. OpenOffice does it right and lets you choose many CSV options when you load rather than assuming anything about the format. $\endgroup$
    – sudo
    Jan 20, 2017 at 17:54
  • $\begingroup$ @sudo The tools depend on your programming language of choice, which is mostly a personal preference. Just to give a few examples, R has been historically a good choice, Python has grown in popularity for data analysis in recent years, Julia is a very promising newcomer in the field. Most programming languages provide mature libraries that give you structures (e.g. dataframes) particularly suited for data analysis and all of them are better than Excel. CSV has been standardized but there are details that are implemented differently, but that shouldn't be a huge issue in your everyday work. $\endgroup$
    – r_31415
    Jan 22, 2017 at 1:27
  • $\begingroup$ I do use Python for light processing, but it doesn't really serve the purposes of Excel. For example, in Excel, you can use tools like autofilter and interactive charts. I usually output my data to a CSV for my higher-ups to look at in Excel or something. $\endgroup$
    – sudo
    Jan 22, 2017 at 2:16
  • $\begingroup$ @sudo Then you want Pandas. Pandas provides a lot of methods to manipulate your data. That includes subsetting based on index, columns or conditions, which is way more flexible and powerful than autofilter. Then you can plot the result ( df.plot()) and export your output to csv (df.to_csv('output.csv')). Keep in mind that data analyses usually require a lot more than filtering and plotting. Therefore, the focus should be on correctness, so you need to decouple presentation from analysis. Perform your analysis in Python (or other language), share your output in csv if that's what you want. $\endgroup$
    – r_31415
    Jan 22, 2017 at 3:38
17
$\begingroup$

Do experienced data scientists use Excel?

I've seen some experienced data scientists, who use Excel - either due to their preference, or due to their workplace's business and IT environment specifics (for example, many financial institutions use Excel as their major tool, at least, for modeling). However, I think that most experienced data scientists recognize the need to use tools, which are optimal for particular tasks, and adhere to this approach.

Can you assume a lack of experience from someone who does primarily use Excel?

No, you cannot. This is the corollary from my above-mentioned thoughts. Data science does not automatically imply big data - there is plenty of data science work that Excel can handle quite well. Having said that, if a data scientist (even experienced one) does not have knowledge (at least, basic) of modern data science tools, including big data-focused ones, it is somewhat disturbing. This is because experimentation is deeply ingrained into the nature of data science due to exploratory data analysis being a essential and, even, a crucial part of it. Therefore, a person, who does not have an urge to explore other tools within their domain, could rank lower among candidates in the overall fit for a data science position (of course, this is quite fuzzy, as some people are very quick in learning new material, plus, people might have not had an opportunity to satisfy their interest in other tools due to various personal or workplace reasons).

Therefore, in conclusion, I think that the best answer an experienced data scientist might have to a question in regard to their preferred tool is the following: My preferred tool is the optimal one, that is the one that best fits the task at hand.

Improve this answer
$\endgroup$
2
  • 6
    $\begingroup$ I would never fault someone for not knowing Hadoop but even in small data situations I feel as if R is superior. There are simply a miriad of things you can do with R that you can't do with Excel. It concerns me this individual has not "discovered" that in his 15+ years $\endgroup$
    – JHowIX
    Apr 3, 2015 at 23:30
  • 1
    $\begingroup$ @JHowIX: Are your familiar with the term "good enough"? I'm also a big fan of R and would prefer it to many tools, Excel included, any day. However, the fact that R can do more doesn't imply that Excel (or any other tool suitable for a task) is inferior in a particular work context. So, while your concern is valid (I refer to that by using word "disturbing"), it might be that the person haven't had an opportunity/need to do that. Remember, that you're talking about time, when R existed, but was popular mostly in academia and data science (termed data analysis or such) was not as hot as today. $\endgroup$
    – Aleksandr Blekh
    Apr 3, 2015 at 23:39
15
$\begingroup$

I think most people are answering without having a good knowledge of excel. Excel (since 2010) has an in memory columnar [multi table] database , called power pivot (which allows input from csv/databases etc), allowing it to store millions of rows (it doesn't have to be loaded on a spreadsheet). It also has an ETL tool called power query allowing you to read the data from a variety of sources (including hadoop). And it has a visualisation tool (power view & power map). A lot of Data Science is doing aggregation and top-n analysis at which power pivot excels. Add to this the interactive nature of these tools - any user can easily drag and drop a dimension on which to break up the results adn I hope you can see the benefits. So yes you can't do machine learning, but I would question how much machine learning is done by data scientists day to day: eg when I want to analyse the prediction errors made in machine learning program I find it easiest to slice and dice the errors with excel.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Interesting. I'm used to the slow and buggy stuff that is Excel 1998-2008. Gotta try out the newer ones. $\endgroup$
    – sudo
    Jan 20, 2017 at 17:58
  • $\begingroup$ I wish I could endorse seanv507's answer a million times. Most of the answers here show that a lot of people are not aware of how powerful the more recent versions of excel are. And please note that when you use the new data analytics tools (E.g Power query, power pivot, DAX) you are no longer limited to 1, 048, 576 rows of data and a host of other limitations without these tools $\endgroup$
    – maze55555
    Jun 28, 2019 at 22:41
  • $\begingroup$ People without a business background don't use excel. Period. And considering business graduates don't usually go into data science, you can understand the ignorance. $\endgroup$
    – NoName
    Nov 5, 2019 at 19:40
  • $\begingroup$ I also endorse seanv507's answer. If a data scientist doesn't know about the features of modern Excel + VBA + Power[Everything] + DAX then they may not be using the right tool for the right job. Many of the complaints listed by Robert Smith can be eliminated by proper training. Just like a person who doesn't know R will do a bad job using R to analyze data, make it hard to reproduce... etc. $\endgroup$
    – Rick Henderson
    Jan 21, 2020 at 2:45
8
$\begingroup$

In his book Data Smart, John Foreman solves common data science problems (clustering, naive bayes, ensemble methods,...) using Excel. Indeed it's always good to have some knowledge of Python or R but I guess Excel can still get most of the job done !

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ Actually, I was quite surprised myself when I read the book that you could do so much with Excel. And that it had evolutionary and other non-linear solvers built-in! A nice benefit of Excel is that your work, especially if you're into reproducible code, is accessible to more people than R or Python code. $\endgroup$
    – Victor Ma
    May 14, 2015 at 23:17
7
$\begingroup$

I'm surprised how many people are attached to the coolness of the profession rather than the actual job to be done. Excel is excellent tool, with free Powerpivot, Powerquery, it can do so much. (these are not available on OS X). And if you know VBA, you can do some nice stuff. And then if you add on the top of that knowledge of python you can combine the very first steps of data extraction and manipulation with python and then use excel, especially if you are a visual person. With excel you can really inspect aggregated data before feeding into any further processes or visualizing. Its a must have tool.

Improve this answer
$\endgroup$
4
$\begingroup$

Let me first clarify that I am starting my journey into data science from a programmer and database developer standpoint. I am not a 10-year data science expert nor a statistical god. However, I do work data scientist and large datasets for a company that works with rather large clients worldwide.

From my experience, data scientist use whatever tools they need to get the job done. Excel, R, SAS, Python and more are all tools in a toolbox for good data scientist. The best can use a wide variety of tools to analyze and crunch data.

Therefore, if you find yourself comparing R to Python, then you're likely doing it all wrong in the data science world. Good data scientist use both when it makes sense to use one over the other. This also applies to Excel.

I think that it's rather hard to find anyone that is going to have experience in so many different tools and languages while been great at everything. I also think it's going to be hard to find data scientist specifically that can not only program complex algorithms but also know how to use them from a statistical standpoint too.

Most of the data scientist I've worked with come in about 2 flavors. Those that can program and those that can't. I rarely work with data scientist that can pull data in Python, manipulate it with something like Pandas, fit a model to the data in R and then present it to management at the end of the week.

I mean, I know they exist. I've read many data science blogs from guys developing web scrappers, pushing it into Hadoop, pulling it back out in Python, programming complex things and running it through R to boot. They exist. They're out there. I just haven't ran into too many that can do all of that. Maybe it's just my area though?

So, does that mean only specializing in one thing bad? No. Plenty of my friends specialize in just one main language and kill it. I know plenty of data guys who only know R and kill it. I also know plenty of people who just use Excel to analyze data because that's the only thing most non-data scientist can open and use (especially in B2B companies). The question you really need to answer is if this one thing is the ONE thing you need for this position? And most importantly, can they learn new things?

P.S

Data Science is not just restricted to "BIG DATA" or NoSQL.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi Glen, thanks for your comments. Take a look at the following link. Its from Swami Chandrasekaran who led the Watson team at IBM, so a pretty experienced data scientist in my opinion. He has programming as basically the third thing a data scientist needs to know, behind "Fundamentals" and Statistics. According to his roadmap, once you know how to program, you are 15% of the way to being a data scientist. Based on this, I might disagree slightly with the statement that true data scientists come in a "non-programming" flavor. nirvacana.com/thoughts/becoming-a-data-scientist $\endgroup$
    – JHowIX
    Apr 6, 2015 at 13:19
  • $\begingroup$ Well, I only say that based on experience. Most statistics and data science courses even don't cover programming outside of what you need for the popular statistical programs. Due to that, most of the guys I run into in the stats world are not good at programming. It's like an afterthought when they enter the real world and realize it helps. $\endgroup$
    – Glen Swan
    Apr 6, 2015 at 13:47
4
$\begingroup$

Excel allows only very small data and doesn't have anything that is sufficiently useful and flexible for machine learning or even just plotting. All I would do in Excel, is stare at a subset of the data for a first glance over the values to make sure I don't miss anything visible by eye.

So, if his favourite tool is Excel, this might suggest he rarely deals with machine learning, statistics, larger data sizes or any advanced plotting. Someone like this I wouldn't call a Data Scientist. Of course titles don't matter and it depends a lot on your requirements.

In any case, don't make a judgement by statements of experience or CV. I've seen CVs and known the people behind it.

Don't assume. Test him! You should be good enough to set up a test. It has been shown that interviews alone are close to useless to determine skills (they only show personality). Set up a very simple supervised learning test and let him use any tool he wants.

And if you want to screen people at an interview first, then ask him about very basic but important insights about statistics or machine learning. Something that every single of your current employees knows.

Improve this answer
$\endgroup$
2
$\begingroup$

Excel can be an excellent tool for exploratory data analysis it really depends on your needs and of course it has its limitations like any tool, but excel definitely deserves a place in the data science hall of fame.

Worth remembering that in practice most users will be exploring a heavily reduced data set anyway (created from an SQL query).

Excel is powerful for exploring data when you use the "table" object in combination with pivot tables, visualising is all 1-2 clicks max and a lot of excel charts in powerpoint look great, unless your looking to create something very bespoke e.g. in a scientific computing context. The interactive nature means you can explore rapidly.

The benefits of the "table" object is that as you transform the data further in excel to enable you to explore new distributions the pivot tables all remember the variable.

Where excel is weak is that the formula list is arguably limiting, for instance a SQL case statement or python statment is way more flexible than an endless chain of if functions.

It really depends on your needs but excel definitely deserves a place in the data science hall of fame.

Interesting anecdote, the team who work on the Facebook newsfeed algorithm can all regularly be seen to be playing with excel and lots of spreadsheets.

Improve this answer
$\endgroup$
1
$\begingroup$

I teach a Business Analytics course that includes SQL and Excel. I teach in a business school so my students aren't the most technically capable, which is why I didn't use something like R, Pandas, or Weka. That being said, Excel is a powerful enough tool to use for some data analysis. It gets most of this power from its ability to act as a front end to SQL Server Analysis Services (a component in SQL Server for data analysis) using the Data Mining Add-In.

SSAS lets you construct decision trees, perform linear and logistic regressions, and even make bayesian or neural networks. I've found that using Excel as a front-end is a less threatening approach to doing these kinds of analyses since they've all used Excel before. The way to use SSAS without Excel is through a specialized version of Visual Studio and that isn't the most user friendly tool out there. When you combine it with a few other Excel tools like Power Query and Power Pivot, you're able to do some fairly sophisticated analysis of data.

Full Disclosure, I'm probably not going to use it again when I teach the new version of the course next year (we're splitting it into two courses so one can focus more heavily on data analysis). But that's just because the university was able to get enough licenses for Alteryx which is even easier to use and more powerful but is $4-85k/user/year if you can't get it free somehow. Say what you will about Excel, but it beats that price point.

Improve this answer
$\endgroup$
1
$\begingroup$

Excel can be an excellent tool. Sure, depending on what you do, it might not fit the bill but if it does, it would be almost foolish to dismiss it. While it takes a while to set up your pipeline, in Excel you can hit the ground pretty much running: built-in UI, easy extendibility via VBA even with Python (e.g. https://www.xlwings.org). It might not be ideal when it comes to stuff like version control but there are ways to make it work with Git (e.g. https://www.xltrail.com/blog/auto-export-vba-commit-hook).

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.