Skip to main content
3 of 3
Commonmark migration

This is a situation that many blogs, companies and papers acknowledge as something real in many cases.

In this paper Data Wrangling for Big Data: Challenges and Opportunities, there is a quote about it

data scientists spend from 50 percent to 80 percent of their time

collecting and preparing unruly digital data.

Also, you can read the source of that quote in this article from The New York Times, For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights

Unfortunately, the real world is not like Kaggle. You don't get a CSV or Excel file that you can just start the Data Exploration with a little bit of cleaning. You need to find the data in a format that is not suitable for your needs.

What you can do is make use of the old data as much as you can and try to adapt the storing of new data in a process that will be easier for you (or a future colleague) to work with.

Tasos
  • 3.9k
  • 5
  • 24
  • 54