We create embeddings to have the dense representation of the sparse feature vectors.
Embeddings map the high dimensional data into a low dimensional space.
A dense feature is the one which has mostly non zeros in it whereas a sparse feature is the one which has mostly zeros in it.
Q: My gut still tells me to make an embedding layer for the movie index as well and concatenate it with the movie-related numerical features. Why though?
Yes, we can do that. Lets say for a particular user "U" we want to pass all the movies he has watched as a feature. We have a total of 10,000 movies in our dataset and the user "U" watched only 10 movies. One way we can achieve this is the "one hot encoding" that is put "1" at the index of the movie which the user "U" watched and put "0" everywhere else. This will create a 10,000 dimensional sparse feature with mostly zero values. This is a significant computation and memory overhead. This is where the "Embeddings" come into play.
"Judge a movie by the company it keeps". As Brian's answers says that the embeddings capture the sematic relationships between entities(movies or users) in our example. There are algorithms which generates embeddings for the input we pass word2vec, glove etc.
If you are curious there is something call "Embedding layer" that is we generate the embeddings with the neural network architecture.
Q: Can't we just pass in the numerical features of the movies for the model to differentiate? Are the embeddings supposed to capture some additional latent information perhaps not captured in the raw metadata?
As per my understanding the embeddings are only useful for categorical features and yes embeddings capture some more information as compare to say a normal categorical feature. Lets think in terms of higher dimensions. When we have simple feature (e.g genre) we need a one hot encoded feature vector to encode the genre of the movie but when we convert this into lets say 2 dimensional dense representation then we can point this feature on a 2D plane. Each point in the 2D plane (x, y) will represent a vector for the movie. Please note that by looking at the point on the 2D we cannot exactly say that what is the genre for that movie it will be just a vector in 2D plane i.e two numbers(x,y). But now we can find the similarity between two movies by computing the distance between their vector representations. This helps the machine learning/ DL model to capture the semantic relationships.
I hope that it was helpful.