Data Exploration Tips & Tools

Question

I am pretty new to the Data Science world and I am looking for some advices.

One of the tasks that I have to do, is to go alone at a customer place for few days/weeks and dig into their data lake to see what it is possible to do with their data.

It is something I have never done before and I am a bit lost on multiple things :

• What are the best methods to find the most interesting data among the other (in a relatively short period of time)
• What tools will I need ? (An Apache Zeppelin with some pre-build dashboards ?)
  • Data Parsing
  • Vizualisation
  • Other ?
• What kind of results is best ? (Cool vizualisation, list of interesting correlation, use cases that could be solved, etc...)
• In this context, is there additional constraints that I may not have anticipated ?

There is various type of customers, with various type of data, so I am looking (if possible) for relatively generic advices.

I have a developer/data-engineer background (Scala, Java, Python), and some basic knowledges in stats and ML.

The goal of my company is to use my results in order to enrich the sofwares that they develop for their customers.

Answer 1

I think you're going about this the wrong way. There's a couple of things that you need to tighten up here:

1. Please don't put yourself out there as a data scientist without a lot more training. When people visit client sites with a lower level of knowledge, it cheapens the profession and only makes it that much harder for people to adopt algorithms in the future
2. You shouldn't start with data. You should start with business use cases for questions that the business is trying to solve. The entire point of data science is to enable you to ask/answer smarter questions. So you need to start with what the business needs, form a hypothesis and then seek out the data you need for your experiments (and that data my not always be completely within your client's domain). You cannot allow yourself to fall into the trap of just looking for "cool" things in the data because they may ultimately not serve any real, long-term purpose to your client. If you have not taken the time to understand the client/business needs then you shouldn't be having any conversations related to data.