Is this a tried alternative to word embedding for NLP?

Question

I'm searching for research related to my idea, but apparently cannot articulate it well enough to the search engines to show me what's been published on this.

My idea: in a deep learning context (text classification), instead of inputting the text as integer vectors representing words in a vocabulary, which get substituted with dense vectors (word embeddings), what if instead, you represent each phrase almost like how a vector of "pixels" represent an image (then use 2d convolutions)? For example, each character, digit, and symbol get an integer value, just like how pixels are represented. So abc becomes 0, 1, 2. The phrase could be formatted as a rectangle at word boundaries or something, but the gist of it, a phrase is formulated as a 2d array where each element is a character from some character set.

Is this a known/tried approach, and what name does it go by? From its apparent lack of popularity, I assume this formulation is far worse than word embeddings, but I'm curious about it.

Answer 1

It sounds like you are talking about a character-level model. Several models consider characters or substrings of words, e.g. FastText [1], but I don't know if any apply convolutions.

[1] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, ‘Enriching Word Vectors with Subword Information’, TACL, vol. 5, pp. 135–146, Dec. 2017, doi: 10.1162/tacl_a_00051.

Answer 2

It has never been tried as far as I know. And the reason is that, IMHO, it would give you inconsistent results. I am talking in particular about unwanted arithmetic properties of such vector.

Do you want to represent an a, b, c vector of letters {1,2,3} knowing that

a < b < c ?

This makes sens with ints, not with char/tokens.

The good news is you can still use 2d conv, once you assembled embedding vectors into larger phrase vectors.

Answer 3

Yes this has been done, in fact there was a period of time before large recurrent models could be reliably trained (and before Transformers were invented) where such methods were state of the art. Paper: "Character-level Convolutional Networks for Text Classification"