I take Natural Language Processing as an example because that's the field that I have more experience in so I encourage others to share their insights in other fields like in Computer Vision, Biostatistics, time series, etc. I'm sure in those fields there are similar examples.
I agree that sometimes model visualizations can be meaningless but I think the main purpose of visualizations of this kind are to help us check if the model actually relates to human intuition or some other (non-computational) model. Additionally, Exploratory Data Analysis can be performed on the data.
Let's assume we have a word embedding model built from Wikipedia's corpus using Gensim
model = gensim.models.Word2Vec(sentences, min_count=2)
We would then have a 100 dimension vector for each word represented in that corpus that's present at least twice. So if we wanted to visualize these words we would have to reduce them to 2 or 3 dimensions using the t-sne algorithm. Here is where very interesting characteristics arise.
Take the example:
vector("king") + vector("man") - vector("woman") = vector("queen")
Here each direction encode certain semantic features. The same can be done in 3d
See how in this example past tense is located in a certain position respective to its participle. The same for gender. Same with countries and capitals.
In the word embedding world, older and more naive models, didn't have this property.
See this Stanford lecture for more details.
Simple Word Vector representations: word2vec, GloVe
They only were limited to clustering similar words together without regard for semantics (gender or verb tense weren't encoded as directions). Unsurprisingly models which have a semantic encoding as directions in lower dimensions are more accurate. And more importantly, they can be used to explore each data point in a more appropriate way.
In this particular case, I don't think t-SNE is used to aid classification per se, it's more like a sanity check for your model and sometimes to find insight in the particular corpus you are using. As for the problem of the vectors not being in original feature space anymore. Richard Socher explains in the lecture (link above) that low dimensional vectors share statistical distributions with its own larger representation as well as other statistical properties which make plausible visually analyse in lower dimensions embedding vectors.
Additional resources & Image Sources: