Skip to main content
organised better
Source Link
Toros91
  • 2.4k
  • 3
  • 16
  • 32
  1. Feels like most of the work is not related to data science at all. Is this accurate?

    Yes

  2. I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that data science requires minimum levels of data accessibility. Am I wrong?

    You're not wrong, but such are the realities of real life.

  3. Is this type of setup common for a company with serious data science needs?

    Yes

From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.

From a practical standpoint, two things:

  • first of all, understand what is technically possible. In many cases, the people telling you no are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the

    First of all, understand what is technically possible. In many cases, the people telling you know are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?

    data types etc?
  • business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...

    Business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...

  1. Feels like most of the work is not related to data science at all. Is this accurate?

    Yes

  2. I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that data science requires minimum levels of data accessibility. Am I wrong?

    You're not wrong, but such are the realities of real life.

  3. Is this type of setup common for a company with serious data science needs?

    Yes

From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.

From a practical standpoint, two things:

  • first of all, understand what is technically possible. In many cases, the people telling you no are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?
  • business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...
  1. Feels like most of the work is not related to data science at all. Is this accurate?

    Yes

  2. I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that data science requires minimum levels of data accessibility. Am I wrong?

    You're not wrong, but such are the realities of real life.

  3. Is this type of setup common for a company with serious data science needs?

    Yes

From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.

From a practical standpoint, two things:

  • First of all, understand what is technically possible. In many cases, the people telling you know are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?

  • Business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...

Include Questions
Source Link
Stephen Rauch
  • 1.8k
  • 11
  • 22
  • 34

Welcome to the real world! :)

  1. Yes

    Feels like most of the work is not related to data science at all. Is this accurate?

    Yes

  2. You're not wrong, but such are the realities of real life in the real world. Not surprised this is hardly ever mentioned in

    I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that data science requires minimum levels of data accessibility. Am I wrong?

    all the sensationalist press articles about how cool data science is

    You're not wrong, but such are the realities of real life.

    and how data scientist is the job of tomorrow
  3. Yes

    Is this type of setup common for a company with serious data science needs?

    Yes

From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.

From a practical standpoint, two things:

  • first of all, understand what is technically possible. In many cases, the people telling you no are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?
  • business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...

Welcome to the real world! :)

  1. Yes
  2. You're not wrong, but such are the realities of real life in the real world. Not surprised this is hardly ever mentioned in all the sensationalist press articles about how cool data science is and how data scientist is the job of tomorrow
  3. Yes

From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.

From a practical standpoint, two things:

  • first of all, understand what is technically possible. In many cases, the people telling you no are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?
  • business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...
  1. Feels like most of the work is not related to data science at all. Is this accurate?

    Yes

  2. I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that data science requires minimum levels of data accessibility. Am I wrong?

    You're not wrong, but such are the realities of real life.

  3. Is this type of setup common for a company with serious data science needs?

    Yes

From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.

From a practical standpoint, two things:

  • first of all, understand what is technically possible. In many cases, the people telling you no are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?
  • business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...
Source Link

Welcome to the real world! :)

  1. Yes
  2. You're not wrong, but such are the realities of real life in the real world. Not surprised this is hardly ever mentioned in all the sensationalist press articles about how cool data science is and how data scientist is the job of tomorrow
  3. Yes

From a technical standpoint, you need to look into ETL solutions that can make your life easier. Sometimes one tool can be much faster than another to read certain data. E.g. R's readxl is orders of mangnitudes faster than python's pandas at reading xlsx files; you could use R to import the files, then save them to a Python-friendly format (parquet, SQL, etc). I know you're not working on xlsx files and I have no idea if you use Python - it was just an example.

From a practical standpoint, two things:

  • first of all, understand what is technically possible. In many cases, the people telling you no are IT-illiterate people who worry about business or compliance considerations, but have no concept of what is and isn't feasible from an IT standpoint. Try to speak to the DBAs or to whoever manages the data infrastructure. Understand what is technically possible. THEN, only then, try to find a compromise. E.g. they won't give you access to their system, but I presume there is a database behind it? Maybe they can extract the data to some other formats? Maybe they can extract the SQL statements that define the data types etc?
  • business people are more likely to help you if you can make the case that doing so is in THEIR interest. If they don't even believe in what you're doing, tough luck...