Since you are specifically asking about deep learning techniques, nothing strikes me out of the bat other than Autoencoders.
You can try using autoencoders for clustering, basically, you need to stick with clustering. Since you don't have the labels.
To answer your question whether they rely on embeddings that are out there, it depends on your data, if you have a domain-specific data or you have the data in a weird language then you should go for creating your own embeddings.
I found this article really helpful, though it has been done on images, you can try it on text, using
Conv1D. Also yes, you can try out word embeddings like word2vec or fasttext. There is this good article where they use gensim for doing attaching the embeddings, in case your data is just plain ordinary English.
there's also this article from keras blog where the author uses pre-trained GloVe embeddings.
Hope this helps.