I am trying to learn how word2vec works to get to more complicated stuff like LSTMs. Because I will use the same training data (so with the same vocabulary) and I want to predict punctuation too, I decided to keep it.

Punctuation is obviously limited to .!,?, anything else is discarded (The sentence itself to avoid loss of meaning). Also, every symbol is converted into a string-like representation to avoid any kind of problems with encoding.

Now, does it make sense to keep punctuation? Should I discard it completely (if I do discard it, how can I generate punctuation for seq2seq model later?)?


You are correct to keep punctuation if you want to be able to predict it.

Tokenization of your input should actually work for any character, be it a letter or punctation. In fact there have already been exmaples of people modelling mathematics using Word2Vec and then generating very realistic maths, via $\LaTeX$ ! Look at the subsection called Algebraic Geometry in Karpathy's now famous blog post.

There is a good note on the matter here, whcih a specific example given in seq2seq learning (basically translation within the realm of NLP). Be sure to read the comments on the accepted answer there.

to answer your final question, I don't think it would be possible to use your generated model to place to required punctuation back into your model, as something like an LSTM would not have a representation for, say a comma, as it had never seen one.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.