1
$\begingroup$

I have a dataset consisting of products, clients, price policy, discounts, quantities, and net sales. The task as put in words by the business is quantity vs price. I have noted a few observations from looking at the dataset :

  1. Discounts: Discounts nullify the effect of any change in the Price policy. So in the end the net sales don't follow this variation. And i observe this for so many client-product pairs.

  2. Seasonality : Variation of quantity for client-product pairs simply follows a seasonality pattern and its not driven by any of the timeseries variables in the dataset. ( I should statistically verify this for now i just did a visual check).

Because at this point i dont see any logic behind how discounts are decided for the clients. Hence there is literally no affect on net sales vs price changes.

How should I model this ? Is this even a machine learning problem because there is simply no causal relation between the variables. If not Price vs Demand then what other things can I propose to the business ?

Edit : 1. Product-client scatter plots UNITARY_NET_SALES VS QUANTITY

enter image description here The first column of the plot shows products are demanded at the same quantities across varying net sales. So no price vs demand effect here.

  1. Timeseries plot for a particular product

enter image description here

Price_list and Discounts have the same behaviour. So whenever the business increases the prices they increase the discounts too, hence the overall affect on net sales is none. And Quantity simply follows a seasonal pattern.

  1. Describe on the columns enter image description here

75% of Quantity is less than 8 units !

Thank you !

Improve this question
$\endgroup$
5
  • $\begingroup$ hard to tell without studying and visualising the actual data $\endgroup$
    – Nikos M.
    Jul 8, 2020 at 17:59
  • $\begingroup$ I'll update my answer with some scatter plots at product-client level, along with timeseries plots. $\endgroup$
    – Siddhant Tandon
    Jul 8, 2020 at 18:34
  • $\begingroup$ Discounts are actually a good indication of price demand curves because they were likely given to increase specific demand (i.e. get the sale). You could try to model discounts and then create a new model on the residuals. $\endgroup$
    – Fnguyen
    Jul 9, 2020 at 16:10
  • $\begingroup$ @Fnguyen could you please elborate on this part then create a new model on the residuals. $\endgroup$
    – Siddhant Tandon
    Jul 9, 2020 at 16:17
  • $\begingroup$ A regression is basically a formula trying to calculate y from x. For n = 1 this formula can be perfect (i.e. using the formula and solving for x gives you exact y), however normally there will be a difference between y_actual and y_regression. This difference is called the residual and normally it indicates regression performance, i.e. low residuals are good. However we can also interpret residuals as all the parts of y that are independent of x or y cleaned of x. So if we use the residuals as the new target value of a model we can identify new relations. $\endgroup$
    – Fnguyen
    Jul 9, 2020 at 16:34

1 Answer 1

Reset to default
2
$\begingroup$

I've tangled with modeling pricing systems over the last two years and one of my key learnings applies here:

Available sales data is often a bad basis for straight-forward prediction tasks and the reason for this is fairly simple:

If you classify all prices (or transactions of a given product at a price x) into "Accepted" and "Not accepted" by the customer you will realize that the data provided by your customer only contains instances of "Accepted" prices.

Therefore a straight-forward modeling of y ~ x with y = demand and x = price is impossible because your y does not vary!

There are several ways around this however. In my comment I mentioned discount being a valuable information!

You have already noticed that discounts are not depended on logical variables, they are seemingly random, this isn't true!

Discounts in most organizations are very, very flexible and often applied manually based on negotiations. This means they are a great indicator of our target y "Acceptance"/"Non Acceptance".

Consider this:

Discount = Demand x Undiscounted_Price

This means that large discounts indicate that the demand is low / negative for the undiscounted price and low discounts indicate demand is high.

To truly discover this relation you might need to model codependent factors and then remove them by training new models on the residuals.

Edit:

An important thing to add especially for the B2B domain. Demand for a certain product is almost set in stone for a customer. Unlike consumers companies do not buy surplus or refrain from purchase due to the prices.

What they do is switch suppliers! This means that there is a really important unknown variable "Customer Demand for product X". You do not want model this variable but you need it to model what you actually want to do:

Share of Wallet or the percentage of the fixed customer demand that was satisfied by your company with the goal being to identify the price that will optimize that percentage. This is helpful to keep in mind because it constrains the performance of any model as you never know whether the historic demand you have in your data is already 0% or 100% of the total demand and thus could not decrease/increase regardless of price.

Improve this answer
$\endgroup$
8
  • $\begingroup$ thanks for the valuable insights. I would spend some days to try your suggestions and maybe comment here again ! $\endgroup$
    – Siddhant Tandon
    Jul 9, 2020 at 17:47
  • $\begingroup$ to summarize the process, I should run 2 level regression, wherein the first level i run a regression on Demand vs Discounts,Prices.... and then second level where Residuals vs Discounts,Prices,.... $\endgroup$
    – Siddhant Tandon
    Jul 13, 2020 at 7:57
  • $\begingroup$ @SiddhantTandon Not quite! The general idea is to identify the relation between prices and demand by exploring the relationship between demand and discounts. To do this it might be necessary/helpful to remove the influence of certain factors by first running a regression of Demand vs. X and then running a new regression of Residuals vs. Z. X in this example could be the discounts or it could be the seasonality. $\endgroup$
    – Fnguyen
    Jul 13, 2020 at 8:09
  • $\begingroup$ @SiddhantTandon For a great example of the process I mean refer to this video: youtube.com/watch?v=go5Au01Jrvs starting from 14:00 onwards he is doing exactly what I said building a regression and then focus on the residuals. $\endgroup$
    – Fnguyen
    Jul 13, 2020 at 8:10
  • $\begingroup$ sorry for troubling you, when you say customer demand for product X you intend that I include customer level features (for every product) and see if it also has some affect on the residuals ? And I did exactly what you suggested, a standard 2 level OLS. For some products the MAE did go down. $\endgroup$
    – Siddhant Tandon
    Jul 14, 2020 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.