Is Excel sufficient for data science?

Question

I'm in the process of preparing to teach an introductory course on data science using the R programming language. My audience is undergraduate students majoring in business subjects. A typical business undergrad does not have any computer programming experience, but has taken a few classes which use Excel.

Personally, I am very comfortable with R (or other programming languages) because I majored in computer science. However, I have the feeling that many of my students will feel wary of learning a programming language because it may seem difficult to them.

I do have some familiarity with Excel, and it is my belief that while Excel can be useful for simple data science, it is necessary for students to learn a serious programming language for data science (e.g., R or Python). How do I convince myself and the students that Excel is insufficient for a serious business student studying data science, and that it is necessary for them to learn some programming?

Edited in response to comment

Here are some of the topics that I will be covering:

• Data processing and data cleaning
• How to manipulate a data table, e.g., select a subset of rows (filter), add new variables (mutate), sort rows by columns
• SQL joins using the dplyr package
• How to draw plots (scatter plots, bar plots, histograms etc.) using the ggplot2 package
• How to estimate and interpret statistical models such as linear regression, logistic regression, classification trees, and k-nearest neighbors

Because I don't know Excel very well, I don't know whether all of these tasks can be done easily in Excel.

Answer 1

First of all check out this post. It has many reasons why Excel is inferior to other solutions, regarding data science tasks. Excel also can't handle large datasets (hundreds of thousands of records - not to mention anything in the vicinity of Big Data), image and sound data.

Excel is good for simple tasks concerning spreadsheets; it emphasizes more on presentation and ease of use, while having minimal support for actually analysing the data. Unless all you want to do is to calculate simple statistical measures (mean, average, etc) or building a very simple model (e.g. linear regression), Excel is inefficient. That being said, 99% the work a company has to deal with concerning data is simple enough to be manageable through Excel.

However Data Science mainly deals with regression, classification and complex models that excel isn't equipped to handle! If your students want to have a look at data science you need to teach them a tool that will be useful to them (R, python, etc.). These languages also have libraries with tons of built in models to "play with".

Another really huge reason I would go with the latter options is that they are open source. I personally feel that open source software should be preferred from an educational standpoint to proprietary solutions (this is also why I suggest python and R over Matlab)!

Answer 2

I just got done with a Masters in Business Analytics and was faced with the same problem you are describing. Luckily I am a technical person and was able to teach myself R and Python, but I was stuck teaching the rest of the class how to use R and Python. The classes I had that used R/Python were handicapped by the lack of technical understanding by the students and so too much time was spent covering how to just open R/Python. The classes that went the other route were underwhelming and not very practical. I wanted to do for a class project something that ended up not being able to be done in Excel because of its limitations but the teacher wouldn't accept any other tools.

It may not be something you can do right away but I would highly recommend that you try and get the department to require a programming course prior to taking your course. Data Science and Business Analytics IMHO should be cross discipline degree paths that require a good bit of Computer Science, but until the programs mature and the university system gets better it might not happen for a while.

Answer 3

I think you need to be teaching them a popular Data Science language like Python or R. Excel is not going to help them in a real job, and isn't practical for data science purposes. I would probably say Python would be most valuable to them in the long run, and with packages like scikit-learn your regressions and classifications can be demonstrated in very few lines of code which they can read and understand more easily. It is not always easy to understand what R is doing by just reading it.

Another word of advice: Don't waste time forcing your students to set up an IDE and download the necessary packages, if you use python create a virtual environment for them with all the necessary packages, and set up an IDE like pycharm(they can get this and most other IDEs under a student/academic license) where then can develop and run their code through UI rather than console which they may find daunting and confusing. If you go down the R route then make sure you have an IDE like RStudio set up for them and make sure all of the includes and package installs are either included in your example code or fully described.

Answer 4

How do I convince myself and the students that Excel is insufficient for a serious business student studying data science

Create in R a huge data.frame (couple mln rows and hundreds of columns), save it as .xlsx.

Show them the time difference in loading it with R, and in Excel on the same machine. Compare basic statistics operations between the two on the same dataset, even plots.

Point no. 2-4 on yout list can be done in Excel too, just A LOT more painfully, show them a couple of example of how much simple (and faster) is filtering with dplyr, compared to basic Excel, again on a huge dataset this would highlight the difference.

Bonus point if you can come up with a dataset that crashes your PC with Excel going.

Also, I'd enphatize the "free-to-use" part of R (or Python). For example, compared to SAS, if you simply want to try one solution (ie some kind of cluster), you load the library, and give it a try, no need to pay more, just for trying.

To me that's the beauty of it, you can try for free whatever you need, and often that's key in DS, imagine if you'd have to pay for each library you install.

Answer 5

Excel and Data Science - sounds really strange to me. Maybe Excel and 'Data Analysis'.

Anyways, I think a good compromise between Excel and R is: KNIME (http://www.knime.org/knime-analytics-platform). It's free on the desktop and much easier to get started. You can import / export to Excel but also use R, Python or Java if the ~ 1.000 nodes miss some functionality that you need. Since the workflows are visually created, it's also much easier to show them to someone who doesn't know any programming languages - which is quite an advantage in some companies.

Answer 6

I think the problem is that you are trying to convince your students that by taking your class, they can do data science similar to the level of modern data science, i.e., fancy stuff like image processing, face recognition. You hear this saying most of the time, "by taking this class, you will..." What you need to teach them is the love for data and the courage to look through a bunch of data, messing around with them to hopefully make some sense out of them. The moment they can do that, you can call them data scientists and you should feel proud of yourself for now having a new generation of data scientists. After that, if they are very serious about data science, they can go on taking other intense courses that deal with math, statistics, and computer science (programming experience like you said). I was in the situation similar to your students. I had no CS background but wanted to break into data science and AI by taking some online classes with fancy promises. I ended up wasting tons of money yet found myself in immense frustration (oh, I need to take this class to know this algorithm, oh they are talking about neural networks now so I have to sign up for the other class, etc.) TL;DR. Tools just account for 1% of the problem you have. With your background, you should have no problem in figuring out the above tasks in Excel in a week.