I am interested in the question, is there any research on the recognition of numbers from audio? For example, there are call records and you need to get the numbers that speak on the phone.
Not aware of research per se but I did this in C# before. The concept you need to know about to recognize only certain patterns in speech is "Grammar".
Grammars in terms of speech or text recognition consist of multiple rules and word tokens that can be identified by your speech recognizer.
As further inspiration for your idea maybe look at SpeechRecognizer and the Grammar class in the VB.net Framework.
This is probably mostly addressed by Speech to Text, and making a text transcription of the whole audio, then afterwards picking out the numbers of interest from the text.
If only considering numbers spoken with a single digit at a time, this can be done with a standard audio classifier, such as in the Tensorflow Simple Audio Recognition tutorial. It uses the Google Speech Commands dataset contains utterances of the words 1-9, along with Yes, No etc.