Statistical knowledge or statistical thinking is useful or necessary to:
Understand, evaluate and pick appropriate metrics to use to evaluate the performance of models.
You need to understand the real-world cost of prediction errors and how each metric relates to this.
Explore and understand the data e.g. to help inform future models or other business decisions or to find and address data errors or anomalies in order to get the best possible performance out of the model.
Investigate and address examples that your model performs poorly on beyond just looking at a handful of examples.
Compare performance of models appropriately by being able to identify when improvements are likely just noise.
If you never e.g. print out a mean or plot a distribution, then I'd be a bit concerned. Although a lot of the ways in which mathematics helps is not about directly calculating some value or whatever, but more about being able to actually understand what you're doing when you're working with large amounts of data and metrics for that data (i.e. statistics).
It also depends on the domain and on which features you're using. If you're doing e.g. image classification, there probably isn't quite as much room for statistical analysis as there would be if you were just doing a classic prediction problem based on distinct and often independent features.