Welcome to the real world! :)
- Yes
- You're not wrong, but such are the realities of real life in the real world. Not surprised this is hardly ever mentioned in all the sensationalist press articles about how cool data science is and how data scientist is the job of tomorrow
- Yes
From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.
From a practical standpoint, two things:
- first of all, understand what is technically possible. In many cases, the people telling you no are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?
- business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...