The problem: I have a huge dataset which is more than 200GBs in size. The dataset contains around 200 columns (predictors). The task at hand is to predict which product to promote to the customer so that he/she ultimately buys it and thus maximizes the revenue of the company.
It is a cross selling situation but I need to look at product recommendation as well as revenue maximization.
From what I understand, revenue maximization would require a regression model. (Correct me if I'm wrong)
I am not sure how to recommend products since there are more than 2000 unique products. (Dummy coding would require tremendous amount of time and resources I feel).
Due to the sheer size of the data, I am planning to use Python for handling the data. (Suggestions about R also welcome)
PS: Forgive me if the problem seems too basic, but I have just started learning
[UPDATE]:
- The data is in long(narrow) format
- I can also use R in order to tackle this
- Products are identified by their unique product ID (2000+ unique product IDs)
- Headers:
date | time | pid | cust_id | ... | amount | tax | net_revenue
- Net revenue: Continuous variable
- Product ID: Continuous but to be treated as nominal
- Cust_id: Continuous but to be treated as nominal