What are some "best practices" for discovering true patterns in data without knowing about them and without relying on "scores" as measures of accuracy?
In university I was always fed data sets that came with instructions or hints as to what models to use. Now by taking some random data set, I've been confused as to how I'm going to discover what kind of model could be the best fit.
"Automatic methods" that use train and test sets seem like an easy way out, because they may work on pretty much any data, but based on experience gained elsewhere, reliance on such methods is often not appropriate, when one envisions also clear interpretability of results. Such methods can make useful predictions on "abstract" data, but I would not use such by default for e.g. medical data. In other words, such models are hard to interpret without comparison, even if all scores look good.
On the other hand, reliance on only G(LM)s might be limited, since I cannot know that the data is linear either. But how am I supposed to know, what other models could fit?
Or is the usage of "score-related" methods the only pragmatic way out of such data sets?
Also, when I learned Bayesian methods I was led to believe that they're the holy grail of predictions, since they deal with distributions and can incorporate also beliefs about the data. But since other models see use, then I'm still confused.
Maybe this pic will help as well: 
We can have a perfectly empirically sound decision tree, but which is low accuracy based on numerical metrics. So how would one even know how to use, e.g., a DNN?
Also how is a score really going to tell about the goodness of the model, since that might depend on the variables chosen?
As a practical example we could consider, for example, this example:
https://www.kaggle.com/code/ashwin4kaggle/drug-classification-data-analysis-modelling/notebook
The patterns here look like multiple approaches could be taken ((G)LM, some train-test method, brms).
