I have a csv with both categorical and float dtypes. I want to do the following:
- For each categorical column i will use pandas to compute the unique values (
pd.unique()
) that are present in the column. sayu_l
for a column - I will use the
len(u_l)
to decide upon the dimension of embeddings that i use for a particular categorical column that i want i embed (this step is the reason i cannot use tensorflow_transform) - I want to create some stateful node that can map
category (token)
value to embeddings index thus subsequently i can lookup the embedding from embeddings matrix that i created in step 2
I dont know how to go about doing it currently. A very inelegant solution i can see is using tensorflow_datasets:
encoder = tfds.features.text.TokenTextEncoder(u_l,decode_token_separator=' ')
- concatenate the entire column using space delimiter (
c_l
) (c_l
is one string now) and then usingencoder.encode(c_l)
This is a very basic thing that i think tensorflow would be able to do relatively easily. Please guide me to the right solution