I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data Engineering from Data Science, on what principles is Data Engineering based, what is the result of the Data Engineering? Is it creating some data structures? If so, what these structures might be? Are there standards or best practices?
$\begingroup$
$\endgroup$
10
-
$\begingroup$ Welcome to DS stack exchange! I think this question might be too subjective for stack exchange. The site focuses on questions that have factual as opposed to opening up a topic for various opinions. Perhaps you could make your question more specific by asking people to provide evidence or share specific experiences and/or why it might be important for people to learn about theory behind it. stackoverflow.blog/2011/01/17/real-questions-have-answers $\endgroup$– fractalnatureCommented Jul 10, 2020 at 5:07
-
1$\begingroup$ @fractalnature Thanks. Please check now, is the question ok? $\endgroup$– MindYBCommented Jul 10, 2020 at 6:27
-
$\begingroup$ what is called data science (or I would prefer machine learning) has certainly much theory behind it. Most importantly a solid understanding of statistics and optimisation. But these are usualy not required in job posts for I think two reasons. 1. Recruiters dont know much about the field themselves, they only include buzzwords hoping to attract the right candidates. 2. If you can hack it and make it work without having solid theoretical knowledge, who cares $\endgroup$– Nikos M.Commented Jul 10, 2020 at 10:20
-
1$\begingroup$ @Nikos M., please note that the question is about Data Engineering without the Data Science component $\endgroup$– MindYBCommented Jul 10, 2020 at 10:39
-
$\begingroup$ I dont think data engineering can be separated from data science. It is like trying to separate applied mathematics from mathematics $\endgroup$– Nikos M.Commented Jul 10, 2020 at 10:41
|
Show 5 more comments
3 Answers
$\begingroup$
$\endgroup$
2
First of all I just want to say that I am not a data engineer and there is definitely someone out there that can answer this better than me.
I do think that there is a lot of theory behind data engineering. It is also very interesting. I too thought that it was boring and I was more interested in just data science/ machine learning. I am not sure if I can say exactly what principles data engineering is based on but it is about how to best store data, access data and creating underlying systems for more efficient computing. The first paper I read that really got me interested in this stuff was the original paper for Spark.
I also just did a quick google for data engineering PhD and came across this. There is a lot of interesting new research going on with how to store data using "nano-structures". There's also an area of research in quantum databases, which seems like a really interesting database abstraction.
I would be interested in hearing a more informed and complete answer from someone else who is in this field! In fact it might be useful to post this question on another stack exchange site.
-
1$\begingroup$ Searching for" PhD for Data Engineering" is a good point! For example IT was once considered part of Computer Engineering or similar curriculums. And programming was once part of mathematics or mathematicaly-related disciplines. The fact that it acquired a name by itself does not make it separate from the field it originates from. Most scientific fields are multi-disciplinary having important connections with many fields. The rationale that scientific fields are isolated is a false one, not warranted by fact $\endgroup$– Nikos M.Commented Jul 10, 2020 at 16:17
-
1$\begingroup$ Yes I agree exactly. Before when I said that they can be discussed separately I didn't mean that I think that it is separate from the field that it originates from. Just that there can be in depth research and theory within the field. I think that Data Engineering is not really a subset of data science but more of an intersection between Computer Science and Data Science. Its like how can comp sci theories (data structures) be used to best support data science use cases. However, I would be interested in someone that might want to write a more in depth answer to this question. $\endgroup$ Commented Jul 10, 2020 at 16:20
$\begingroup$
$\endgroup$
2
There certainly is theory, or at least competing methodologies, behind ETL and Data Warehousing, for a start look at the Inmon vs Kimball methodologies.
In a nutshell (I could talk for days on this subject), Bruce Inmon's (the Father of Data Warehousing) methodology revolved around building a large, loosely 3rd normalized data warehouse from multiple sources, that business domain-centric reporting star-schemas could be quickly built and disposed of as needed, whereas Kimball concentrated on (through some staging steps) building directly into reporting structures.
In my experience, whilst the Inmon philosophy looks the more sensible, Inmon based projects, at least those I've been involved with, tended to fail a lot more than Kimball based ones, primarily due to the time and effort required to build the large Data Warehouse before any business value can be seen.
There is a lot more to it, and I've probably let my own experience and opinions taint the purity behind of the methodologies (you can google for larger discussion), but I mention it largely to illustrate that, even in the simple (hah) process of moving and consolidating data, many a religious war has been fought :) Also be aware that most of my practical DW experiences were about a decade ago, so the field has probably moved on.
-
$\begingroup$ Is data warehouse or a star schema a structure that can be a prerequisite to the data science step? $\endgroup$– MindYBCommented Jul 11, 2020 at 6:05
-
$\begingroup$ @MindYB Yes and no, it depends on your requirements and what you are sourcing from. I was more responding to the ETL in your question with my ETL experience. If you are looking at real-time business data analytics (large, realtime data feeds) or even batch updated data, then, yes, you are going to have to have a properly structured Data Warehouse (with Star Schemas, Dimensions, Fact tables and the like) $\endgroup$– johncCommented Jul 23, 2020 at 2:56
$\begingroup$
$\endgroup$
The book, Designing Data Intensive Systems by Klippman, does an excellent job of building up the data engineering stack starting from their needs, instead of just introducing tools one after another.
I am not sure if that is what you mean by theory, but if you get a good grasp of the different phases of data (in storage, in transit, being processed, being collected and being distributed etc.) which you can, by reading the book, that should give you the solid foundation to connect with and use effectively newer tools as they come along, and to architect effective systems, choosing the right tools.
