# Would keeping all punctuation make any sense in word2vec?

I am trying to learn how word2vec works to get to more complicated stuff like LSTMs. Because I will use the same training data (so with the same vocabulary) and I want to predict punctuation too, I decided to keep it.

Punctuation is obviously limited to .!,?, anything else is discarded (The sentence itself to avoid loss of meaning). Also, every symbol is converted into a string-like representation to avoid any kind of problems with encoding.

Now, does it make sense to keep punctuation? Should I discard it completely (if I do discard it, how can I generate punctuation for seq2seq model later?)?

Tokenization of your input should actually work for any character, be it a letter or punctation. In fact there have already been exmaples of people modelling mathematics using Word2Vec and then generating very realistic maths, via $\LaTeX$ ! Look at the subsection called Algebraic Geometry in Karpathy's now famous blog post.