this is my first post at ds StackExchange, so please be gentle and let me know if something is not clear :)
I have many products (>1M), and I save all the products purchases in a DB with a time stamp. ("purchases data")
each product has 'content features' (e.g product size, product safety rank etc.)
The "purchases data" looks like this:
| time stamp | product id | content features 1 | ... | content features N |
Where each row is a purchase of a product with id at time stamp.
My main target is to get tomorrow most wanted products, I translate the problem into predicting demand for the next day, or classify each product id and day as high demanded or low demanded),
I struggle with two main problems with these settings:
- Generating demand data: I want to convert the "purchases data" into demand for a day ("demand data")- meaning that I group the data by product id and day,
Then I count the number of rows and save it as 'freq' (and also remove row duplication).
The problem is that the minimum product frequency per day would be 1 and not 0
For example: if product #1 was purchase at Sunday 3 times and at Wednesday 2 times, the purchases and demand data would be:
"purchases data" fi(product id) is mapping to content feature i:
| time stamp | product id | content f 1 | ... | content f N | | Sunday 05:20 | 1 | f1(1) | ... | fn(1) | | Sunday 08:11 | 1 | f1(1) | ... | fn(1) | | Sunday 10:25 | 1 | f1(1) | ... | fn(1) | | Wednesday 08:10 | 1 | f1(1) | ... | fn(1) | | Wednesday 16:20 | 1 | f1(1) | ... | fn(1) |
| day | product id | content f 1 | ... | content f N | freq | | Sunday | 1 | f1(1) | ... | fn(1) | 3 | | Wednesday | 1 | f1(1) | ... | fn(1) | 2 |
But if product #1 was not purchase at Monday there wouldn't be any row since there is no purchase data for this item at that timestamp.
Since there are over 1M products, I want to avoid creating rows with 0 frequency,
Is there a way to create (or to design) demand data from purchases data for a huge amount of items (products) without using 0 demand rows?
- Content-based time series: After creating "demand data", I want to use it as a time series.
My problem is that I would need to split the data into over 1M series, one for each product/item id, and these series would also be very sparse... I want to find a way to use the "content features" as input with the time series and have the model learn to use some kind of averaging of time series based on related content features.
What is the best way to model content based series time prediction?