0
$\begingroup$

I've found out that most of the choices made during model selection are based on a sort of trial and error. From what I've heard, even the most experienced Data Scientists cannot know beforehand whether a model will perform better than the other in a given set of data.

If that is indeed the case, how can I better explain architectural/model choices I make?

For instance, if I am presented with a set of data and asked what should be used, how can I avoid using the phrase "because my intuition says so?"

Ideally the kind of answer I would be looking for would not just tell me which models use in which case, but how to defend these choices. A reference explaining, for instance, the construction of the map linked would be the closest to what I would be looking for.

$\endgroup$

1 Answer 1

0
$\begingroup$

Most of the work done in data science is experimental in some way (from data acquisition to a model being used to do something), partially due to how most techniques work (hyperparameters, distributional assumptions, etc.), after all most models are very tied to the task they were built to solve, so they don't translate very well even to tasks that are very similar from human perspective, since they are directly dependent on data.

There is also important aspects to consider like model size, hardware constraints (you would find difficult to embed GPT-3 anywhere), intrinsic task problems that come from data (like unbalanced classes in a classification task, outliers and missing data in any task), interpretability, etc..

Thus architectural choices themselves are subject to some of these restrictions, specially when you consider that performance is not linear with the structures used in the architecture, be it number of neurons, connections or whatever else, in other words, using less things might get better results than more things.

Now answering your question more directly, intuition is never used as justification, robust and rigorous experiments are. One may start with models or values that are found on literature about similar problems, but eventually exhaustive experimentation will get to the best stuff so far and if a particularly rigorous investigation scheme was used, the architecture found may be close or even be the best for a long time.

The link to the scikit flowchart displays more about what scikit can do for you in specific circumstances than what is possible to do, most of it comes from experience, but is in no way final on what is possible.

There are environments where getting more examples is too expensive or even impossible, there are also domains where concept shift exists, so your models need to be able to be flexible and adapt to change in distributions, don't take these flowcharts/guides as the absolute truth, because there is not a single tool or toolkit that does everything, they are, however, a good starting point.

Usually, at least in my experience, we just apply Occam's Razor and eventually use the minimal model or models (ensembles is a thing) that can get the job done because there are plenty of other concerns and constraints that are as important as solving the problem itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.