8
$\begingroup$

I am about to start a job in which I will be working with large datasets and will be expected to find trends, etc... I have found lots of resources on where to learn ML and other hard skills and feel that I am (semi) competent on this end.

I am interested in knowing if there are specific soft skills that are helpful as a data scientist. What are things you wish you knew starting out?

While Kaggle is very useful when learning, it also presents clear objectives. How do you handle being given a dataset, but no clear objective?

Let me know if this is too broad, I can think of more specific questions.

$\endgroup$
  • 3
    $\begingroup$ tip #1: never pet a burning dog $\endgroup$ – Brandon Loudermilk May 31 '16 at 19:54
  • 1
    $\begingroup$ If you do not mind, please share the industry you are in. The maths and the concepts remain the same, however the structure of the data varies and also how one may approach it. The below advises are very apt and if practiced are going to be a big help. I hope by knowing the industry I may be able to share something that you can directly relate. $\endgroup$ – Drj Jun 1 '16 at 0:35
  • $\begingroup$ I hope whoever interviewed you for this job is now reading this and thinking "why didn't we ask those questions at interview?". $\endgroup$ – Spacedman Jun 1 '16 at 7:13
  • $\begingroup$ Drj, I will be working in part with data from the manufacturing process and in part with customer feedback data. It seems like a broad spectrum. I'm coming from academia where the data was produced by my own experiments and I had very clear goals. $\endgroup$ – Hobbes Jun 1 '16 at 15:47
7
$\begingroup$

I think there are a lot of important soft skills to consider in the Data Science domain.

Here are some of them:

  1. Know for a fact what the goal is, spending a lot of time on data wrangling, models, visualization and reports when it was not all for the specific goal in mind is a waste. Communicating with less technical people is a skill in itself.
  2. Iterate repeatedly with the product owner. Keep making sure you are on the right path.
  3. If the data doesn't tell the story they thought/want tell them it is not the case, be clear in why this is happening, what biases might be playing a role etcetera. Do not apply all kinds of filters or keep changing parameters to get the desired results.

Regarding your second question:

The objective has to be either gotten from the product owner explicitly or derived from a less mathematical objective. An example could be where you need to predict train arrivals based on some features. They want the model to predict as many times as possible within a 10-minute error range. This is relatively explicit.

Sometimes it is less clear than that, they might say we need it as accurate as possible. Then you will have to decide what to optimize, in some cases, this will just be minimizing the MSE but in other cases, other things might make more sense for your case. Usually, this will be clear from the implicit objective and something that you will get better at with more experience. Both implicit and explicit objectives derive from clear communication with the product owner.

$\endgroup$
  • $\begingroup$ Thanks for the comment, I think your advice about communicating with less technical people is really helpful and definitely something I need to work on. $\endgroup$ – Hobbes May 31 '16 at 16:41
  • $\begingroup$ I added some info about the objective too $\endgroup$ – Jan van der Vegt May 31 '16 at 16:55
  • $\begingroup$ Very helpful, I will keep this in mind moving forward. (Guess I can't up-vote until I have a higher reputation) $\endgroup$ – Hobbes May 31 '16 at 19:18
5
$\begingroup$

"How do you handle being given a dataset, but no clear objective?"

This will be common.

Apart from the advice above, understand that it is essential to understand the goals of the business you are in, and of your immediate client. Frequently you will need to understand the specific problem that made them turn to data better than they do. It is very highly common to be presented with data and an unclear objective from your internal or external client - it will be usually your task to supply a goal that can be achieved with the data and will solve the client's actual business problem. An amount of lateral thinking will be required to make the data outcome and business solution match.

I would summarise the above as 'defining the objective is too important (and possibly too difficult!) to be left to the client (alone)'.

In the machine learning context, CRISP-DM is a methodology which tries to solve this problem by iterating through a loop so that additional data understanding can be used in discussion with the client to better understand the original problem. So, for example, they may state a ill-defined goal, a second discussion after you've done some EDA will sharped it a little. When you later produce a model that works well, but isn't on quite the right target, you'll get closer to the real business goal again.

In other words, don't be too disturbed by the fuzziness of the task. Expect to encounter a vaccuum, and fill it to your advantage.

It's a slight sideways shift, but the six sigma methodology attempts to solve this problem in a different context with the DMAIC system (the 'D' standing for 'Define', in terms of the 'voice of the customer'), so it is probable that some tips can be gleaned in resources for the six sigma context (e.g. exercises you can do with a client that help them express what you want more clearly)

$\endgroup$
  • $\begingroup$ Thanks, that's great feedback. I especially like 'defining the objective is too important (and possibly too difficult!) to be left to the client (alone)'. I will definitely look into CRISP-DM. $\endgroup$ – Hobbes Jun 1 '16 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.