I've seen several job descriptions for data science which include developing a novel algorithm to be a part of production environments. Can you give some input of what could be meant here exactly? Would they mean an algorithm that behaves somewhat of an ETL: getting the data, cleaning it, storing it, and running a known model on it? Or something more complex like building variations of known prediction algorithms? Some examples will really be nice, as I'm studying to go into this field.
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Could you give an example? $\endgroup$– Juan Esteban de la CalleJan 17, 2020 at 19:09
-
1$\begingroup$ @JuanEstebandelaCalle here's one: wnwd.com/careers/data-scientist $\endgroup$– MariahJan 17, 2020 at 19:12
-
1$\begingroup$ From the looks of it, it sounds like a position in a R&D lab for someone with a PhD. They do exist but that's not necessarily representative of all data scientist positions. $\endgroup$– RelaxedJan 19, 2020 at 14:03
-
$\begingroup$ "Algorithm" can be used broadly sometimes. Throw project-specific data into an off-the-shelf GB Random Forest, and some people will refer to that as a "novel algorithm". $\endgroup$– AcccumulationJan 20, 2020 at 19:13
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject online. My opinion:
New algorithms are developed in research centers and at universities. And even then, most algorithms used in companies are already developed, and just optimized even more. Don't get your hopes up or be afraid that you have to reinvent gradient descent backpropagation.
With developing algorithms they most likely mean data extraction, data cleaning, data preparation for reporting statistics and presenting graphs. Maybe programmatically, maybe just using tooling.
The presented data may give more insights in simple relations in the domain, and maybe insights in more complex questions that can be asked.
You may get to define data flows, get to compare and select machine learning algorithms and tune its parameters. And continuously evaluate model performance in practice.
$\begingroup$
$\endgroup$
In industry its usually variations (but important ones) of the ground ideas.
Look at this boosting timeline:
- (Ada)Boosting Formally by two profesors in 2003
- xgboost by DLMC Distributed Machine Learning Community in 2014
- lightgbm by Microsoft in 2017
- catboost by yandex in 2017
- +- all the variations in between that did not caught up
Building on "basic" idea they removed all the negative ones while modifying/defining new sequence of steps to be executed.
To answer your question. As long as you have some non-trivial (debatable what it is) new (or variation of) sequence of steps to be executed, with appropriate degree of generalisation you got youself an algorithm. So the type of new algo is dependable on the field you work in.
answered Jan 18, 2020 at 8:24
$\begingroup$
$\endgroup$
It is indeed extremely rare for someone to develop a novel algorithm to solve their problem. In my experience it is more important to understand the business domain, how to normalize the data and choose what loss function should be minimized.
But it is very valuable to have experience with various kinds of algorithms so that you can pick the right tool for the job.
If a job listing says that a person "must develop new algorithms" I'd read it more like "must develop new programs / software / scripts".
$\begingroup$
$\endgroup$
You might be interested in the annual Kaggle survey about the state of Machine Learning and Data Science.
Some key results related to your question:
The most commonly used algorithms were linear and logistic regression, followed closely by decision trees and random forests. Of more complex methods, gradient boosting machines and convolutional neural networks were the most popular approaches.
Some numbers from the report (2020):
Linear or Logistic Regression - 83.7%
Decision Trees or Random Forests - 78.1%
Gradient Boosting Machines (xgboost, lightgbm, etc.) - 61.4%
Convolutional Neural Networks - 43.2%
Bayesian Approaches - 31.4%
Recurrent Neural Networks - 30.2%
Neural Networks (MLPs, etc.) - 28.2%
Transformer Networks (BERT, gpt-3, etc.) - 14.8%
Generative Adversial Networks - 7.3%
Evolutionary Approaches - 6.5%
$\begingroup$
$\endgroup$
1
None.
What you do in business is program algorithms to solve biz problems.
Most all common problems have algorithms and processes already. So what you would do is adopt and apply it to your particular situation.
Data science being the buzzword du jour, and very poorly understood by almost everyone as to what it is and how to use it, is more a fancy word to get you to apply your programming skills to databases in some fashion.
Often this will be some sort of data 'mining' that looks for patterns so that management can ASSume that correlation means causation and then use that as a rule to make decisions.
This is how the last wall street meltdown occurred. They made BAD ASSumptions that were close enough for most cases to ignore the errors in it but when that black swan hit they went bankrupt and the government had to bail them out with our tax money.
-
1