4
$\begingroup$

I am trying to find best practices for scaling data science teams, i.e find an efficient workflow/methodology to divide work between Software Engineers and Researchers working on a same product.

I’ll explain: both the SE and Researchers need the output produced by the others but they don’t necessarily have the same constraints.
- What’s important for a SE is: code maintainability, testing, CI/CD, refactoring codebase for improved development velocity,l as little branches as possible in the repository
- What’s important for a Researcher is: pace of experiments, management of experiments, journaling of experiments, model management and versioning, multiple Git branches for experimentation

How can we reconcile between both when working on the same Git repository, in a way that satisfy both stakeholders and make the work as efficient as possible ?

For example, researchers may be unhappy about a significant refactor of their experimentation script to a package that break down code to smaller bits of code, or may be frustrated by having to make sure their code does not break existing CI tests.

Can you think of interesting patterns (or point to interesting resources, books, blogs etc.) that help make the process smoother when both stakeholders are working in the same team / product ? Thanks much.

$\endgroup$

1 Answer 1

1
$\begingroup$

One potential solution to reconcile the different needs of software engineers and researchers on a data science team is to use a separate branch in the Git repository for experimentation. This way, researchers can work on their experiments and model development in a separate branch without affecting the main branch that is used by the software engineers for code maintainability, testing, and deployment.

To ensure that the work of the researchers is integrated into the main branch in a way that does not compromise the code quality and maintainability, a code review process can be implemented. In this process, the researchers can submit their experimental code for review by the software engineers, who can provide feedback and suggestions for improvements before the code is merged into the main branch.

Another potential solution is to use a package-based development approach, where the code for experimentation and model development is organized into modular packages. This can help improve code maintainability and make it easier for both software engineers and researchers to work on the codebase without disrupting each other's work.

Overall, it is important for the data science team to establish clear communication and collaboration processes to ensure that the different needs of the software engineers and researchers are addressed and that the work is carried out efficiently.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.