I've been playing around trying to teach my self some basic data science with datasets I have access to work, I've just finished prepping a 46 million row dataset which I wrangled using pandas and sql alchemy to store my MSSQL dB.
Imagine the table as follows :
Date | Sale | Location | SKU | Sale | Quantity | InvoiceID
now what I'm trying to do is prep my data to follow this guide by Chris Moffit on practical business python, my first instinct was to use Python only as I'm not that great with SQL, but the data was to large, so I had to learn some basic sql. I've now loaded my data into SQL with the correct metatypes.
now, what I'm having trouble with is the amount of data I have, my SKU data for a month is over 25k unique items, even when I remove outliers, and set a min value for the article to be over, say £10, it's still quite difficult.
I wonder if I'm not understanding this correctly but trying to use this on such a large data set?
When I attempt to wrangle this to match the sample feature set in the link above, I get an index out of range error, and if I try to create a crosstab in SQL, I hit the max 4k or so columns (using MS SQL server).
am I using the wrong tools or I am just woefully out of my depth?