Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in Chinese text since native speakers can easily segment words from text. In order to assist language learning tools, it's helpful to have an automatic way to segment words within blocks of text.
Some approaches use dictionary based greedy algorithm methods but are prone to failing due to the common problems that most greedy algorithms present.
I want to try using neural networks, but my question is, How would one go about encoding the characters to the input neurons of the network?
I am not talking about OCR. The characters are already known and encoded in unicode, but how would I present the characters to the inputs of the network?
One method I can imagine is to have the network look at the text in portions of say 100 characters at a time, and have one neuron for each character. But how would I represent the character as a number to the network? Using the unicode integer number value for the character doesn't seem like a good idea.