3
$\begingroup$

The problem: I have a huge dataset which is more than 200GBs in size. The dataset contains around 200 columns (predictors). The task at hand is to predict which product to promote to the customer so that he/she ultimately buys it and thus maximizes the revenue of the company.

It is a cross selling situation but I need to look at product recommendation as well as revenue maximization.

From what I understand, revenue maximization would require a regression model. (Correct me if I'm wrong)

I am not sure how to recommend products since there are more than 2000 unique products. (Dummy coding would require tremendous amount of time and resources I feel).

Due to the sheer size of the data, I am planning to use Python for handling the data. (Suggestions about R also welcome)

PS: Forgive me if the problem seems too basic, but I have just started learning

[UPDATE]:

  • The data is in long(narrow) format
  • I can also use R in order to tackle this
  • Products are identified by their unique product ID (2000+ unique product IDs)
  • Headers: date | time | pid | cust_id | ... | amount | tax | net_revenue
  • Net revenue: Continuous variable
  • Product ID: Continuous but to be treated as nominal
  • Cust_id: Continuous but to be treated as nominal
Improve this question
$\endgroup$
2
  • $\begingroup$ How can you "predict which product to sell to the customer"? You can only try and sell it. Do you mean that? Do you want to figure out which product to promote in the hope that the customer will buy it? Or have they already decided to buy something? Or what? $\endgroup$
    – Spacedman
    May 10, 2016 at 7:20
  • $\begingroup$ Yes, recommendation of a product. The product that is most suited to the customer based on 200 predictors which would maximize my revenue. I have historical data containing their past purchases based on which I should recommend them new products. $\endgroup$
    – Anonymint
    May 10, 2016 at 7:25

3 Answers 3

Reset to default
1
$\begingroup$

If I understand your question correctly, you need to recommend one product out of 2000. You want to pick the one with the highest revenue expectation. This can be decomposed in the probability of making the sale times the revenue given that is a sale. The revenue for a sale is known, the probability is what is needed. I assume you have historic data of which customers (with the given features) bought which of these products. Now you can train regression models to estimate the probability of a sale for a specific product. Logistic Regression and Neural Networks with a sigmoid activation functions are well suited for unbiased probability estimates, by optimizing the log-loss cost function.

With this amount of data I propose to start on a subset of your data and using Logistic Regression. Since this is a generalized linear model you will likely need to do some feature engineering to increase performance. Since you mentioned you use Python, scikit-learn has a lot of functions for Logistic Regression optimization and feature engineering so take a look at that.

Once you have trained 2000 models, for a new recommendation you would run the features through the models, get all the probabilities, multiply them with revenue corresponding to this product and pick the highest one.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I can break down the whole problem into two parts, one is product prediction and the other is revenue maximization. I can use logistic regression to predict the probability of recommending a given product but for that wouldn't it require the product variable to be dummy coded? (since products are nominal in nature) $\endgroup$
    – Anonymint
    May 10, 2016 at 11:55
  • $\begingroup$ Yes you would need to dummy code them, and then split them up per product column, where some of the entries will be 1 and most 0, then train your regressor on this column as target $\endgroup$
    – Jan van der Vegt
    May 10, 2016 at 12:30
  • $\begingroup$ Dummy coding will result in creation of 2000+ new columns over my existing ones. How do I go about it in R or Python? I have a RAM limitation of 8GB $\endgroup$
    – Anonymint
    May 10, 2016 at 12:34
  • $\begingroup$ You don't need to create them at the same time, just iterate over your set for product 1, make a 1/0 column, train your model and export it. This will be very computationally expensive but that will be the case regardless with this amount of data $\endgroup$
    – Jan van der Vegt
    May 10, 2016 at 12:53
  • $\begingroup$ You could use sklearn's stochastic gradient descent classifier (sklearn.linear_model.SGDClassifier) with the log loss function. That way you don't have to load everything into memory, instead just feed batches to it. Or have a look at vowpal wabbit, it can train logistic regression models from terabytes of data.. on a single machine. $\endgroup$
    – stmax
    May 10, 2016 at 13:31
0
$\begingroup$

Your question right now is really vague. However, if I had to make a recommendation based on what you have written I would suggest doing a market basket analysis to determine which products are commonly purchased together. Once you understand which products are purchased together you can do a regression analysis of the combos found in the market basket analysis in order to predict revenue maximization

Improve this answer
$\endgroup$
0
$\begingroup$

Depending on the volume and density of the information you have about your customers and their purchases, you may be able to run a collaborative filtering, meaning finding the cluster of similar customers and see what each cluster tend to buy.

Then, for each individual you can recommend what she has not bought before and has a greater metric (revenue in this case).

From the business standpoint, there may be other metrics you may also want to take into account (net profit, time in stock, etc.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.