# Why do we need data scientists if there are websites like this? [closed]

Very basic question about data science as I'm a beginner. I noticed that there are sites like illustrated below, that help visualize and analyze data so that the business owner can have a better understanding of their business.

My question is that if there are sites like this, why are data scientists needed? What can they offer that sites like this can't?

• You would need data scientists in the first place to create such website and ultimately for maintaining it. Jan 27 at 20:18
• It's the equivalent of asking "why do we need doctors since there are sites with medical information?" Try curing your own cancer from medical websites ;) Jan 27 at 21:22
• Nice point by Erwan, which is itself a similar answer by doctors when asked about models which can help diagnose diseases :) Domain experts is, and will be, needed Jan 27 at 21:40
• They answer the question themselves: "Get business insights in 5 minutes". Dumping a 50GB dataset (which a data scientist would call small!) into that website over your 1Gbps internet connection would take more than 5 minutes. A simple machine learning algorithm would take hours or days to crunch your data, and possibly give you a dozen results which you can then investigate to find out how to tweak the algorithm, possibly by coding in R and C and adjusting the ETL layer written in Python. Yes, that would a typical scenario for a data scientist. What the website offers...is not. Jan 28 at 10:41
• Nice ad framed as a question. Jan 28 at 11:53

Without having a complete knowledge of the features on that website I would say:

• Data visualization is only one part on data scientist (ds) pipeline from data understanding thought model validation and model production as per CRISP-DM, and that site seems to be focused only on that visualization part.

• Introductory courses on data science most of the time work with "ready to use" data frames (Iris, Titanic) that do not reflect the real way in which data must be preprocessed and aggregated to the correct level (ex. you may have accounts level data that needs to be aggregated to user's level to predict user's default) also, in many cases you have multiple sources of data stored in a variety of places like relation tables or non structured information that will need to be queried and joined beforehand.

The latter role is commonly attributed as a "data engineer", however in small to medium projects the line between the two roles are less discernible. It is important to note, however, that this line should be fuzzy, as more often than not choices made by data scientists require work from a data engineering perspective, - as other answers below have touched more upon.

And this gives a wrong impression of simplicity on the task.

Besides, the above step would need for you to have at least basic understanding of the data you are working on in order to create "meaningful" features that otherwise will end just by aggregating by min, max and average (sometimes useful but not sufficient)

• Finally, a trained model in a notebook would hardly being helpful if you cannot use that model in an completely deployed way (having a AWS lambda for example)

It is a fair question, and now more than ever before with tools like Datarobot and alike... In addition to the answer by Julio Jesus, being the point about dataset building step one of the most important ones and time consuming, some other relevant points are:

• the selection of right evaluation metrics for your models is crucial for a right interpretation of your models, and that also depends on your specific use case

• not only selecting a right predefined metric, but also sometimes you even have to create a custom metric to be used for your model training

• labelling process: it is very important to carry out a confident labelling of each sample (when doing supervised learning) and is, usually, not straightforward

• specific business case data cleaning and processing, taking into account that not all types of basic filtering methods can be applied automatically and basic imputation strategies of missing values might be not enough (from a data quality perspective) to get a real use case sense. A nice reference for this point is a recent tweet by Andrew NG mentioning another tweet on this topic by François Chollet.

• also about data visualization, these tools usually offer a limited predefined list of options, which you can extend more flexibly if you, as a data analyst/scientst/whatever expertise, know how to code it, interpret it and explain it

In general this kind of site can only propose standard methods applied to standard data. This can be pretty useful, but it covers only a small proportion of the vast diversity of problems addressed by data science.

In a broad sense, data science has applications in virtually every imaginable domain (e.g. medicine, astronomy, self-driving cars, machine translation...) and with every possible kind of digital data (e.g. text, speech, images, video...). The scope of this kind of website is limited to one domain, typically solving standard business problems using standard business data. It's easy to see that the world is full of non-standard problems just by browsing a few questions here on DataScienceSE.

I work in forecasting retail sales, e.g. predicting next week's sales of a particular stock-keeping unit in the presence of promotions, price changes, day of week, calendar events, seasonality and tons of other drivers. (I used to tell my kids that daddy makes sure there is always enough ice cream at the supermarket.)

Often, the retailer will wonder just why a particular forecast was off. Why was a promotion underforecast so badly, or the Christmas sales overforecast? To answer that, and to improve forecasts going forward (so we don't have too much product clogging the shelves, or spoiling in the case of perishables, or too little product and unhappy customers), you need to understand the data, and the model, and understand if the model could be improved. Or you may need to help the customer understand that the model did the best it could, and that there is simply a lot of residual variation. In which case the question becomes one of how best to deal with this residual variation, by using higher safety stocks, or consciously allowing for stockouts. At this point, business logic enters.

Also, when we have a new retailer ramping up with our forecasting product, we need to map their promotion landscape to our model. Retailers can have quite complex promotions (buy $$n$$ units of product $$X$$ and $$m$$ of $$Y$$, then you get $$p$$ units of $$Z$$ at $$x\%$$ off, and $$y$$ bonus points on your shopper card, and $$z$$ airline miles...). Again, you need understanding here. AI is not quite there yet.

Related, though closed: Data science without knowledge of a specific topic, is it worth pursuing as a career? My answer there focused on the necessity of communication and business understanding to a data scientist, both of which a website won't provide.

In my opinion, you need to understand the models, plots, calculation routines to be able to draw the right conclusions out of it.
There is absolutely no use to have, let's say a Q-Q-Plot, and lack the statistic knowledge to interpret it. A few other examples:

• "cleaning" data for nice and round-up plotting requires either good knowledge of the data or else, if done automatically, assumes statistical properties such as normality, non-collinearity etc. An automatic algorithm either introduces a large bias by transforming the data or will give you a false impression or relations/correlations etc.
• F.i. when checking for (multi-)collinearity of data, you can use the correlation coefficient. This is typically the Pearson-correlation. But Pearson is only for linearly related data. Now you most often have non-linearly related data, so Pearson coeff. is low and you say: Ok, no collinearity. Instead, with proper statistic knowledge, you can select the correct coefficient and then extend the analysis with other measures such as VIF, condition index, partial pairwise correlation etc...
• also to know what you can get out of data visualization, you need to know at least the basic set of data visualization methods. F.i. when you want to analyse sales dependig on the size of your stores. But now all your large stores are in low income regions. Plotting a scatter plot of it you see no relation/correlation. So your assumption is: There is no use to build larger stores. But in fact, the interaction between store size and average regional income is concealing the "real" influence. You need to know which methods are good to find this "hidden" information and how to apply them. And, in my opinion, finding this "hidden" information is what discerns a real data scientist from the nowadays everyone-is-a-datascientist-hipster
• predictions are highly biased towards preprocessing of input data, the type of the model and the model parameters. You almost never find the best model on the first try.
• last but not least: If something is "super robust" you can be 99% certain, that some assumptions have been made, introducing bias, cutting outliers which are not outliers, applying overly strong regularizations/transformations etc... If something is super robust: Either know why it is super robust and have the comparison to non-"super"-robust, or be overly skeptical about the resulting data quality. Robustness is never for free, so you should better know the backgrounds of it.

In my opinion, the problem is that everyone wants to be a data scientist, but no one wants to learn statistics. So we've got tons of data scientist university courses and degrees of some sub-prime universities doing data science name-dropping, but now covering the statistic background. So everyone is applying something but barely anyone knows what he/she is doing.

It has come so far, that I've seen two "real" statistic oriented data scientists with a master in computer science or engineering, both having a quite solid statistical background and working with data science on a day-to-day basis (but have no degree explicitly stating "Data Scientist"), being told when applying for a data scientist job: "You've got a too low experience with data science, we've hired someone with a bachelor degree in data science." From which you know exactly: Those hiring guys have no idea about data science and they just want to get "something" done... Well... This is Germany and we are known for having an extremely unflexible job market... ;)

So: Learn your statistics or stop "being" a data scientist. :)

The exact role of a data scientist varies incredibly from organisation to organisation; o the validity of this answer varies on exactly how you define data scientists.

But I would say that the output of a data scientist is not a visualization; It is the answer to a question.

A data scientist is asked a question like: "why were online sales down in March?" They dive into the data and try and answer it. So a possible answer is: "Our web site was really unresponsive in March, and pages kept timing out" And they will support that, and discour that by looking at and showing visualiations. E.g. that historically when sales have been down it also has corresponded with increased latency, and that this was occuring also in March.