how to predict content based demand

Question

this is my first post at ds StackExchange, so please be gentle and let me know if something is not clear :)

I have many products (>1M), and I save all the products purchases in a DB with a time stamp. ("purchases data")
each product has 'content features' (e.g product size, product safety rank etc.)

The "purchases data" looks like this:

| time stamp | product id | content features 1 | ... | content features N |

Where each row is a purchase of a product with id at time stamp.

My main target is to get tomorrow most wanted products, I translate the problem into predicting demand for the next day, or classify each product id and day as high demanded or low demanded),

I struggle with two main problems with these settings:

1. Generating demand data: I want to convert the "purchases data" into demand for a day ("demand data")- meaning that I group the data by product id and day,
   Then I count the number of rows and save it as 'freq' (and also remove row duplication).
   The problem is that the minimum product frequency per day would be 1 and not 0

For example: if product #1 was purchase at Sunday 3 times and at Wednesday 2 times, the purchases and demand data would be:

"purchases data" fi(product id) is mapping to content feature i:

   | time stamp      | product id | content f 1 | ... | content f N |
   | Sunday  05:20   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  08:11   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  10:25   | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 08:10 | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 16:20 | 1          | f1(1)       | ... | fn(1)       |

"demand data":

   | day             | product id | content f 1 | ... | content f N | freq |   
   | Sunday          | 1          | f1(1)       | ... | fn(1)       | 3    |     
   | Wednesday       | 1          | f1(1)       | ... | fn(1)       | 2    |  

But if product #1 was not purchase at Monday there wouldn't be any row since there is no purchase data for this item at that timestamp.

Since there are over 1M products, I want to avoid creating rows with 0 frequency,

Is there a way to create (or to design) demand data from purchases data for a huge amount of items (products) without using 0 demand rows?

2. Content-based time series: After creating "demand data", I want to use it as a time series.
   My problem is that I would need to split the data into over 1M series, one for each product/item id, and these series would also be very sparse... I want to find a way to use the "content features" as input with the time series and have the model learn to use some kind of averaging of time series based on related content features.

What is the best way to model content based series time prediction?

Answer 1

Welcome to the site. I would encourage you to think about your problem in a different way. You are focused on "what sold today" whereas you should be focused on "who bought what over a historical timeline".

What you're looking for is known as a recommender system and there are (generally speaking) two types:

1. Content-based - what should you recommend based on attributes of products. The algorithm is basically saying, "You bought breakfast cereal, here are other products that might go with your cereal . . ."
2. Community-based - what should you recommend based on attributes of people who bought products. The algorithm is basically saying, "You are a female, under 30, with no kids. Other females, under 30 with no kids also liked these products . . ."

I will assume that you don't have info on your customers so let's focus on the content-based recommenders. You are on the right track by thinking about the attributes of products, but you should be thinking about them both (1) over a longer timeline than just yesterday and (2) how the products and their attributes relate to each other. The people who need attribute X might also need attribute Y and that is (most likely) across multiple products and will generate higher demand for those products.

Start researching content-based recommender systems in your language/tool of choice and you will end up with the desired algorithm. From there you can also think about user data collection and then move into a community-based recommender over the long term.