What are the basic principles/tools necessary to make something like Alexa utterance parsing?

For reference, Alexa allows a designer to define phrases with "placeholders" that will be filled in. For example, the phrase "what is the horoscope for gemini" would match the underlying model (below), and return Sign=gemini.

what's the horoscope for {Sign}
what is the horoscope for {Sign}
give me the horoscope for {Sign}
tell me the horoscope for {Sign}

To clarify: I am interested in the theory behind how the language model and parsing (with regard to theory and algorithms) work, so I can build my own version.

  • $\begingroup$ I wrote an echo app a few months ago so my knowledge is rudimentary, but believe the actual speech-to-text is done at the device level (i.e. the physical echo/alexa unit). Text is then sent to a remote process, where it attempts to match the utterances in the skill set. Obviously placeholders like {Sign} function as arguments to functions whose values are supplied by the end-user/speaker. $\endgroup$ – Brandon Loudermilk Mar 6 '16 at 18:41
  • $\begingroup$ Thanks @BrandonLoudermilk. I updated the question to clarify that I am trying to understand the natural language parsing aspects. $\endgroup$ – Matt C Mar 6 '16 at 19:10
  • $\begingroup$ Speech to text and text to intent (NLP) are done on Amazon's servers; not in the cloud. I was also wondering the same thing. $\endgroup$ – Timmmm Apr 7 '16 at 7:08

If you're interested more generally in speech understanding or speech-to-text, some approaches to natural language parsing and speech-to-text use recurrent neural networks or Hidden Markov processes for learning, as well as a number of signal-processing algorithms to extract more data from the input stream that just raw audio. Keep in mind people have spent their whole careers on this work, so it's not a good problem to just pick up and run with unless you're a MS/PhD candidate looking for a capstone/dissertation project. Here's the iconic paper from Bell Labs that inspired a lot of the DFA/HMM solutions. I've yet to find a paper that does a good job of explaining how to actually implement the RNN style solutions, but here's one in case you're interested.

It's likely that Alexa uses some combination of these methods, but I doubt you're going to get any good answer out of anyone here. After all, it's an important Amazon project and it's not like their engineers are going to come on Stack Overflow and start giving away trade secrets.


Curiously, a half way authoritative answer is possible. I too wanted to know this answer. Googling found very little until I stumbled across a Data Scientist job posting on amazon's web site. At least as of August 28, 2017 they were looking for:

Solid background in statistical learning techniques for NLP (HMMs, CRFs, SVMs, LDA, LSI, MRFs etc)

So this isn't an exhaustive list, in particular there could be interesting technologies that they don't want to advertise, but the above list is at least a subset of what techniques they use.

  • 1
    $\begingroup$ It's an interesting list, but unless the job is exclusively working on Alexa's language routines (please add that detail to the answer if true), then beware that Amazon - like many other big internet companies - is applying data science in many areas of its business, and a Data Scientist role at Amazon could have nothing to do with Alexa. $\endgroup$ – Neil Slater Oct 9 '17 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.