Can machine learning models treat a vector as a whole feature to learn

Question

We know a ML model naturally takes a feature vector with real valued elements as input and learn to predict. But can it treat a fixed-size vector as a whole feature to learn?

For example, when using a 128-bit string as a feature, we can calculate the total number of different strings occurred in datasets and apply one-hot encoding to them. However, the dimension of the encoding can be very large（thousands）in this way. Or we can use binary to represent the 128-bit string, which only contributes to 128 dimensions. I don't know if the traditional ML models (not CNN) such as random forests are able to learn the dependence of the 128 features. Or is there any ways they can treat the 128 features as a whole?

Answer 1

The underlying question is: what is a feature, what does it represent, and how can a ML algorithm use it?

A feature is an indicator, it's supposed to help the algorithm predict the response variable. So the semantic of the feature is crucial: for instance it's easy to see that a patient's age can be a relevant feature for detecting a particular disease, whereas knowing their last name isn't.

What can a ML algorithm do with a feature? It just compares it: only equality/difference tests (boolean, exactly what happens with one-hot encoding) for categorical features, order tests for numerical features.

But can it treat a fixed-size vector as a whole feature to learn?

Per my last point, the algorithm needs a way to compare the "whole vector". As far as I know this can only happen:

• either by comparing the whole vector to any possible value it might take: this is a boolean test, we go back to the one-hot encoding case.
• or by comparing every individual cell independently, which is the idea of using the binary representation described by OP:

Or we can use binary to represent the 128-bit string, which only contributes to 128 dimensions. I don't know if the traditional ML models (not CNN) such as random forrests are able to learn the dependence of the 128 features.

In this case the problem is about the semantic of these individual cells: what do they represent and can they help predict the response variable?

Take for instance a spam binary classifier: traditionally with one-hot encoding every feature represents whether a particular word appears in the text document. This makes sense, because knowing whether the document contains for example the word "viagra" or not gives an indication about whether it's spam or not. Now if you just use the binary representation of the word "viagra", the feature $i$ represents whether the $i^{th}$ bit is 1 in the ASCII or UTF8 binary representation of the word. Quite clearly this is not a relevant indicator to decide whether the document is spam or not (btw this can be tested with correlation or other measures). So sure one can technically provide the individual bits as features, however the predictions will be near random because the model doesn't have relevant indicators as features.

Important note: I focused on the semantic issue so I didn't even talk about the problem of whether order matters, but this would also be an issue.

Answer 2

To avoid having too much features produced by a OneHot, you can either :

• Transform your data before your OneHot, regrouping different classes in intervals you created yourself (for example, reducing your 128 features to 30 classes you created yourself) then applying OneHot
• Use CategoryEncoder (Such as TargetEncoder, James-Stein, LeaveOneOut, WeightOfEvidence) : this will make stats and transform the classes into a number, higher if the class is well known (appearing often) and tend to give more True prediction than the average