Convolutional neural network for sparse one-hot representation

I have some basic features which I encoded in a one-hot vector.

Length of the feature vector equals to 400. It is sparse.

I saw that conv nets is applied to a dense feature vectors.

Is there any problems to apply conv nets to a sparse feature vectors?

I would not apply convolutional neural networks to your problem (at least from what I can gather from the description).

Convolutional nets' strengths and weaknesses are related to a core assumption in the model class: Translating patterns of features in a regular way either has a minor impact on the outcome, or has a specific useful meaning. So a pattern 1 0 1 seen in features 9,10,11 is similar in some way to the same pattern seen in features 15,16,17. Having this assumption built in to the model allows you to train a network with far fewer free parameters when dealing with e.g. image data, where this is a key property of data captured by scanners and cameras.

With one-hot encoding of features, you assign a feature vector index from a value or category essentially at random (via some hashing function). There is no meaning to translations between indices of the feature vectors. The patterns 0 0 1 0 1 0 0 and 0 0 0 1 0 1 0 can represent entirely different things, and any associations between them are purely by chance. You can treat a sparse one-hot encoding as an image if you wish, but there is no good reason to do so, and models that assume translations can be made whilst preserving meaning will not do well.

For such a small sparse feature vector, assuming you want to try a neural network model, use a simple fully-connected network.

• "...use a simple fully-connected network" or any classifier for that matter. Jan 24 '17 at 10:57

Although I agree with Neil Slater's response, you should keep a couple of things in mind.

1) "you never know!" In data exploration, you never know what you may find. If you have a ton of data, perhaps playing around with a 20x20 conv net will give you some decent results. Of course, it would be helpful if there are more than just a few features for it to learn...if your 400 length vector is the result of one-hotting 4 different features then it's probably safe to say that a conv net won't give you much.

2) If you're looking for a reason to implement a conv net, then go for it. Even if your accuracy metrics are terrible you at least get to learn how to create your net, train, and predict using your own data...one cannot underestimate this learning experience! So much more valuable that running yet another mnist example out of the box.

3) Comparison. Make a regular net and a conv net...then you get to compare the two. Not only that, compare it to a Random Forest, logistic regression, etc. etc.. Do this enough times and you start to develop intuition.

I say do it! (unless somebody is paying you...in which case try the regular NN first)

• I'm sorry but I can't see how in this case you can even think of trying to train a convnet, it simply can't answer this kind of question for the reasons Neil Slater explained : features vectors do not have in general a "spatial" meaning (except special cases e.g. SIFT). For example, you can permute dimensions. On an image, where you have spatial structure, you scan for patterns and convnets are one of the many possibilities. Anyhow, in this case it just doesn't make any sense and moreover it's wrong. Modulo the fact that the question is a little bit vague. May 19 '15 at 21:35
• @BertrandR yes, as I said, I agree with Neil...likely it will not work. And yes, the question is a bit vague...most likely because nub may not know much about NNs and CNNs. But (s)he has data and sometimes you learn lessons much better by working with your own data. Even getting a net to train can be an accomplishment when you're starting out. I'm not saying to use a NN or CNN because I think it will give the best result...I'm encouraging nub to experiment and break stuff and learn. May 20 '15 at 6:39
• Don't agree too much with this answer. There is only so much time to spend in exploration: the point is to make it count. This answer does not support that CNN would be a likely best fit for the OP's scenario. Dec 11 '15 at 17:03
• @javadba Once again, more voice of reason. Perhaps my answer should have been in the form of a comment. My point was to encourage going through the process. If he's looking for a reason to learn, using one's own data can be motivating. Dec 13 '15 at 5:43