Tips for a new data scientist

Question

I am about to start a job in which I will be working with large datasets and will be expected to find trends, etc... I have found lots of resources on where to learn ML and other hard skills and feel that I am (semi) competent on this end.

I am interested in knowing if there are specific soft skills that are helpful as a data scientist. What are things you wish you knew starting out?

While Kaggle is very useful when learning, it also presents clear objectives. How do you handle being given a dataset, but no clear objective?

Let me know if this is too broad, I can think of more specific questions.

Answer 1

I think there are a lot of important soft skills to consider in the Data Science domain.

Here are some of them:

1. Know for a fact what the goal is, spending a lot of time on data wrangling, models, visualization and reports when it was not all for the specific goal in mind is a waste. Communicating with less technical people is a skill in itself.
2. Iterate repeatedly with the product owner. Keep making sure you are on the right path.
3. If the data doesn't tell the story they thought/want tell them it is not the case, be clear in why this is happening, what biases might be playing a role etcetera. Do not apply all kinds of filters or keep changing parameters to get the desired results.

Regarding your second question:

The objective has to be either gotten from the product owner explicitly or derived from a less mathematical objective. An example could be where you need to predict train arrivals based on some features. They want the model to predict as many times as possible within a 10-minute error range. This is relatively explicit.

Sometimes it is less clear than that, they might say we need it as accurate as possible. Then you will have to decide what to optimize, in some cases, this will just be minimizing the MSE but in other cases, other things might make more sense for your case. Usually, this will be clear from the implicit objective and something that you will get better at with more experience. Both implicit and explicit objectives derive from clear communication with the product owner.

Answer 2

"How do you handle being given a dataset, but no clear objective?"

This will be common.

Apart from the advice above, understand that it is essential to understand the goals of the business you are in, and of your immediate client. Frequently you will need to understand the specific problem that made them turn to data better than they do. It is very highly common to be presented with data and an unclear objective from your internal or external client - it will be usually your task to supply a goal that can be achieved with the data and will solve the client's actual business problem. An amount of lateral thinking will be required to make the data outcome and business solution match.

I would summarise the above as 'defining the objective is too important (and possibly too difficult!) to be left to the client (alone)'.

In the machine learning context, CRISP-DM is a methodology which tries to solve this problem by iterating through a loop so that additional data understanding can be used in discussion with the client to better understand the original problem. So, for example, they may state a ill-defined goal, a second discussion after you've done some EDA will sharped it a little. When you later produce a model that works well, but isn't on quite the right target, you'll get closer to the real business goal again.

In other words, don't be too disturbed by the fuzziness of the task. Expect to encounter a vaccuum, and fill it to your advantage.

It's a slight sideways shift, but the six sigma methodology attempts to solve this problem in a different context with the DMAIC system (the 'D' standing for 'Define', in terms of the 'voice of the customer'), so it is probable that some tips can be gleaned in resources for the six sigma context (e.g. exercises you can do with a client that help them express what you want more clearly)