1
$\begingroup$

So my question is more on the conceptual side.

Given a dataset, I want to predict a given continuous variable Y. Now, there are 3 features, 2 categorical and one numerical (integer only). I know that if I create a combination of the 2 categorical features I can use the numerical feature as an independent variable in the linear regression model to predict Y. For instance, for a combination of the 2 categorical features I build a linear model where the numerical feature is the independent variable. This yields good results because the relationship between Y and the numerical feature for a given combination will always be linear. However this also means that I might have to build 1000 linear regressions, one per combination. This of course sounds a bit strange to me, since I can use for instance a decision tree model without the need of creating the combinations of the categorical features. I'm trying to see the pros and cons of each methodology, but I'm having a hard time. Can anyone shed some light into this problem?

Example:

Imagine this dataset is from a massive bakery. The dataset is very large, over 100k instances. The categorical values are a machine ID and the recipe that it does. So the combinations are (machine, recipe). And the numerical feature is the number of times you do the recipe. Therefore I'm trying to predict the how long a given process will take. Now obviously, for every combination, the more you do the recipe, the more time it will take. The thing is, it seems super odd to just create a linear equation for every combination. Sure it works, but you end up with a enormous amount of linear equations which seems like it will take a lot more computational power than one single model.

$\endgroup$
3
  • 2
    $\begingroup$ What is $y$ and $x$ here? I suppose you want to include the categorical features as $x$ (independent variables). Did you check „dummy encoding“. The question/problem is not clear to me. $\endgroup$
    – Peter
    Nov 14, 2021 at 23:22
  • 1
    $\begingroup$ I think there is some confusion about the terms: features = independent variables = usually called x = the input information for the model. target = dependent variable = usually called y = the output that the model predicts. $\endgroup$
    – Erwan
    Nov 14, 2021 at 23:36
  • $\begingroup$ Features are not the same as independent variables here. Categorical values, unless encoded cannot be input of a linear equation, therefore, they are not independent variables in a linear equation. And that is the case here, these categorical values cannot be encoded. I edited the question to add some context. $\endgroup$
    – DPM
    Nov 15, 2021 at 0:04

1 Answer 1

1
$\begingroup$

In general the choice between a single model and multiple independent models depends on the expected level of dependency between the variables.

In your example, if two different machine ids implies two completely distinct machines with different characteristics, and therefore completely different relation between the recipe id and the duration, then it makes more sense to train a distinct model for each machine.

On the other hand if a lot of information is common between different machines and recipes, for example if the time for recipe R on machine M can be deduced from the time of recipe R on some other machines, then a single model could leverage this kind of correlation.

  • An important factor to decide is the number of instances: the case of multiple independent models requires that you have a sample large enough (and representative enough) for every model.
  • In case there are enough instances for every model, it's likely that the independent models would provide more accurate predictions since the model wouldn't conflate all the data.

At the end of the day, the only sure way to know which one is the best option is to run the experiments.

$\endgroup$
1
  • $\begingroup$ Thank you for your answer, this analysis is what I was missing. I was trying to do an assessment of the problem before hand and then compare both methodologies to see if the conclusions match. Thank you very much! $\endgroup$
    – DPM
    Nov 15, 2021 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.