I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just define a dictionary or mapping function before hand.
Say for example that I am using food labels - my current data has the following labels:
['Steak','Potatoes','Soup'], but it is possible that later, I will gate data with the labels
'Chow mein', and I have no way of knowing the list of all potential labels before hand. Moreover it is possible that some of the incoming labels are proper names or strings that are idiosyncratic and not part of any standard vocabulary, e.g.
'Double Super Mac-Whopper'.
I thought of simply building my own hash map, but then I would have to put a lot of effort into saving and versioning the resulting map to maintain consistency across experiments and later in production.
I tried using the
int.from_bytes function in Python 3, but it gives wildly varying int sizes (I think because it is using string length):
> int.from_bytes('Bok Choy'.encode('utf-8'),'little')
I looked at the sklearn categorical encoders (
sklearn.feature_extraction.FeatureHasher), but they all seem to require knowledge of the number of categories before hand (by having to specify a dictionary or fitting an encoder to the available data, etc...)
I thought about using some word embeddings like word2vec, but they return pretty large vectors, and all I need is an int, and I don't really care about semantic similarity etc...(i.e. using a word embedding is overkill).
Is there some sort of preprocessing utility from and ML library, or some publicly available string to int hash map that is stable that I can use?