Training word embeddings does not rely on optimizing the cosine similarity of words. It usually relies on prediction problems. Take for instance the skipgram model: you are predicting the context of a word, given the word. Such models, project words in geometric spaces (for example the commonly used ~300 dimensions). In other words, a word is associated with 300-dimensional dense vectors. Due to the way that these vectors are learned, they capture the semantics of the words and hence similar words are close to the induced space.
Cosine similarity (or dot product) captures exactly this semantic closeness. Intuitively, we want similar words to be close, because they are similar and we hope that the space models some of the word properties and meaning.