I need to merge data from 1000s of excel files provided by different operations managers on productivity and other reports. The excel files have similarity of data but the headers are all custom since being different manager and different clients.
For example manager A will have a.xlx and manager b will have a.xlx, but the headers for each will be different though the data inside will be usually the same. Each day 100 different excel files are updated by all team members via new files e.g a_todays_date.xlx and manager b using /a.xlx.
Is this something that can be handled via python ML libraries.
What is the best way to merge all these data and save to the DB on a per day basis. The average data would per day would be around 15GB. The end goal is to create dashboard.