In my work, we collect sales data of our products. We have a set of 1st level customers (lets call that group as jacks) with whom we do we business. These jacks then sell our products to end customers (let's call that group as roses).

These sales data contain fields such as product,product type, market segment (like APAC, EMEA etc),qty requested (by jacks),order booked qty (by jacks),cost price (our selling price to jacks),revenue (our expected revenue based on qty requested and price) ,date of purchase order, sold price (jacks selling price to roses).

Please note that order requested variable is captured before itself (during initial negotiation) but just appended to this csv

Now, with this data, our business objective is to increase our revenue (which means the order booked qty by jacks (for them to sell it to roses) should be high).

However, in real time we see that there is a huge discrepancy between order requested and order booked quantities.

So, if we can predict order booked qty in advance and if it is found to be less, we can make sure that we don't over produce or follow up with the jacks to place more orders (or know reason why aren't they booking enough orders).

Should I do linear regression for this? But my data contain repeated measures. Meaning, a customer from jack group can appear multiple times. Meaning, once he would have sold product A where order requested and booked kind of match (less difference) but the same customer would have sold product B where the difference between order requested and order booked is really huge (affecting our revenue and unnecessary backlog in our inventory).

Can linear regression work for repeated measures?

As you can see each customer may or may not have multiple records. So, which algorithm would be better for this scenario?

If you think this problem has to be framed differently, do let me know please


1 Answer 1


It's a quite complex problem, there are certainly many ways to look at it.

A simple non-ML option that comes to mind is to just use the history of the "jack" and the product: for every jack and/or every product, calculate for example the average amount of "over-requesting" in the past. If this average is higher than some threshold, flag the next request as a potential "over-request".

Now as a regression problem, as far as I understand, it's necessary to consider both the jack and the product as independent variables. So I don't think that the same jack with two different products would count as a repeated measure. However in the history there might be some instances with same jack and same product, once with "over-requesting" and other not. This would be more like a repeated measure as far as I know, but anyway it doesn't really matter: linear regression or any other regression method can deal with inconsistencies in the data, as long as there is some general pattern. But I doubt that linear regression would be flexible enough for the variations in this kind of data, I'd suggest something decision tree regression (M5P) or SVM regression (SVR).

  • $\begingroup$ Thanks Erwan for your help as usual. Upvoted. $\endgroup$
    – The Great
    Commented Dec 16, 2021 at 10:11
  • $\begingroup$ I just thought about it. Regarding repeated measures. Since each row (also called as transaction) are different (have a unique transaction Id), can it be considered as non-repeated? Meaning, a customer from Jack group would have sold same product twice but on different dates for different price. So, we need to find the prediction for each record. So, then am I right to assume that it is not repeated? $\endgroup$
    – The Great
    Commented Dec 22, 2021 at 4:04
  • $\begingroup$ @TheGreat honestly I don't know whether this would count as a repeated measure, alI can say is that your explanation makes sense to me. But more importantly I don't think that repeated measures would be a serious problem in general, although this would certainly depend on the precise design and method used. $\endgroup$
    – Erwan
    Commented Dec 23, 2021 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.