Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [mixture-of-experts]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers

How does a spare mixture of experts route each token separately to an expert?

As far as I can understand the routing mechanism in the the sparsely-gated mixture-of-experts layer routes each token to its own expert. Considering that the output of the attention layer is a three ...
gamer bg's user avatar