How to go about creating embeddings (especially, token to Id mapping) for categorical columns in tensorflow 2.0+?

Question

I have a csv with both categorical and float dtypes. I want to do the following:

1. For each categorical column i will use pandas to compute the unique values (pd.unique()) that are present in the column. say u_l for a column
2. I will use the len(u_l) to decide upon the dimension of embeddings that i use for a particular categorical column that i want i embed (this step is the reason i cannot use tensorflow_transform)
3. I want to create some stateful node that can map category (token) value to embeddings index thus subsequently i can lookup the embedding from embeddings matrix that i created in step 2

I dont know how to go about doing it currently. A very inelegant solution i can see is using tensorflow_datasets:

1. encoder = tfds.features.text.TokenTextEncoder(u_l,decode_token_separator=' ')
2. concatenate the entire column using space delimiter (c_l) (c_l is one string now) and then using encoder.encode(c_l)

This is a very basic thing that i think tensorflow would be able to do relatively easily. Please guide me to the right solution

Answer 1

Embedding() layers take sequences as inputs. If there is no time dependency in your data (such as for time series datasets or language corpora) then those layers cannot be used. What kind of data do you have?

Alternatively, you can use dimensionality reduction, one-hot encoding or, in some cases, label encoding. I personally suggest dimensionality reduction whenever possible, even if it's the most computationally intensive choice.