this is my first post at ds StackExchange, so please be gentle and let me know if something is not clear :)

I have many products (>1M), and I save all the products purchases in a DB with a time stamp. ("purchases data")
each product has 'content features' (e.g product size, product safety rank etc.)

The "purchases data" looks like this:

| time stamp | product id | content features 1 | ... | content features N |

Where each row is a purchase of a product with id at time stamp.

My main target is to get tomorrow most wanted products, I translate the problem into predicting demand for the next day, or classify each product id and day as high demanded or low demanded),

I struggle with two main problems with these settings:

  1. Generating demand data: I want to convert the "purchases data" into demand for a day ("demand data")- meaning that I group the data by product id and day,
    Then I count the number of rows and save it as 'freq' (and also remove row duplication).
    The problem is that the minimum product frequency per day would be 1 and not 0

For example: if product #1 was purchase at Sunday 3 times and at Wednesday 2 times, the purchases and demand data would be:

"purchases data" fi(product id) is mapping to content feature i:

   | time stamp      | product id | content f 1 | ... | content f N |
   | Sunday  05:20   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  08:11   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  10:25   | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 08:10 | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 16:20 | 1          | f1(1)       | ... | fn(1)       |

"demand data":

   | day             | product id | content f 1 | ... | content f N | freq |   
   | Sunday          | 1          | f1(1)       | ... | fn(1)       | 3    |     
   | Wednesday       | 1          | f1(1)       | ... | fn(1)       | 2    |  

But if product #1 was not purchase at Monday there wouldn't be any row since there is no purchase data for this item at that timestamp.

Since there are over 1M products, I want to avoid creating rows with 0 frequency,

Is there a way to create (or to design) demand data from purchases data for a huge amount of items (products) without using 0 demand rows?

  1. Content-based time series: After creating "demand data", I want to use it as a time series.
    My problem is that I would need to split the data into over 1M series, one for each product/item id, and these series would also be very sparse... I want to find a way to use the "content features" as input with the time series and have the model learn to use some kind of averaging of time series based on related content features.

What is the best way to model content based series time prediction?


1 Answer 1


Welcome to the site. I would encourage you to think about your problem in a different way. You are focused on "what sold today" whereas you should be focused on "who bought what over a historical timeline".

What you're looking for is known as a recommender system and there are (generally speaking) two types:

  1. Content-based - what should you recommend based on attributes of products. The algorithm is basically saying, "You bought breakfast cereal, here are other products that might go with your cereal . . ."
  2. Community-based - what should you recommend based on attributes of people who bought products. The algorithm is basically saying, "You are a female, under 30, with no kids. Other females, under 30 with no kids also liked these products . . ."

I will assume that you don't have info on your customers so let's focus on the content-based recommenders. You are on the right track by thinking about the attributes of products, but you should be thinking about them both (1) over a longer timeline than just yesterday and (2) how the products and their attributes relate to each other. The people who need attribute X might also need attribute Y and that is (most likely) across multiple products and will generate higher demand for those products.

Start researching content-based recommender systems in your language/tool of choice and you will end up with the desired algorithm. From there you can also think about user data collection and then move into a community-based recommender over the long term.

  • $\begingroup$ thanks for answering, i dont have any information about the customers, they are all new customers, and i want to get an image about the products demand regardless of the costumer attributes, my target is to rank tomorrows products based on probability of demand and i dont see how a recommender would help me in this task, correct me if im wrong, but when using a content based recommend ill need to submit a product as input and the recommender would recommend some similar products regardless of the demand trend.. $\endgroup$
    – data_3
    Feb 20, 2019 at 11:53
  • $\begingroup$ @Sharon I’m clear on both of those. But I’m encouraging you to think about how the products relate to each other and the recommendations behind each. Ranking the products isn’t necessarily an algorithmic problem. Quite frankly that’s just some applied statistics and relatively straightforward. The rankings gain you very little. Besides, how will you know that milk is rising because cereal is popular? With your proposed approach you are just looking at products in a silo and assuming all demand is just sales driven per product. $\endgroup$ Feb 20, 2019 at 11:59
  • $\begingroup$ I appreciate your input and i hope i understood your approach, i quite agree on using straightforward statistic, that is what im doing now, but im asking if there is a way to predict demand based on time and content, creating a recommendation for each user is another issue that im trying to avoid right now.. $\endgroup$
    – data_3
    Feb 20, 2019 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.