# Visualizing word embeddings

I am working on a text-classification problem, and trying to understand how to work with the tensorboard projector for an embeddings layer in Keras. Borrowing an example from the Deep Learning with R book, I have a model set up like this:

library(keras)

max_features <- 2000
max_len <- 500

imdb <- dataset_imdb(num_words = max_features)
c(c(x_train, y_train), c(x_test, y_test)) %<-% imdb
x_train <- pad_sequences(x_train, maxlen = max_len)
x_test = pad_sequences(x_test, maxlen = max_len)

model <- keras_model_sequential() %>%
layer_embedding(input_dim = max_features + 1, output_dim = 128,
input_length = max_len, name = "embed") %>%
layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>%
layer_max_pooling_1d(pool_size = 5) %>%
layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>%
layer_global_max_pooling_1d() %>%
layer_dense(units = 1)

summary(model)

model %>% compile(
optimizer = "rmsprop",
loss = "binary_crossentropy",
metrics = c("acc")
)


When it comes time to configure the callback_tensorboard object, however, I'm a little lost. It seems the API has changed since the book was written, and I haven't found a good working example yet. The embeddings_data property is apparently required if the embeddings_freq parameter is set, and it needs to match the shape of the model inputs c(?, 500). I can satisfy this by simply passing all the word tokens as a matrix:

tensorboard("my_log_dir")

callbacks = list(
callback_tensorboard(
log_dir = "my_log_dir",
histogram_freq = 1,
embeddings_freq = 1,
embeddings_data = matrix(1:max_features, nrow = 4)
)
)

history <- model %>% fit(
x_train, y_train,
epochs = 5,
batch_size = 128,
validation_split = 0.2,
callbacks = callbacks
)


The model trains OK, but when trying to look at the result in projector, it seems that the embeddings are being processed as 64,000 dimensions (2,000 words * 128d layer output?) with a single point for each 500 element vector in embeddings_data. This makes PCA computation stall out.

I expect a final result closer to the example at https://projector.tensorflow.org/, where Dimensions are equal to the layer output dimensions, and there is a single point for each word.

What am I missing? Are there any good working examples of visualizing a standard word embedding layer with current versions of Keras and Tensorflow?

I am using R 3.3.4, Keras 2.2.0, and Tensorflow 1.11.

It seems there are a couple issues at work here.

First, apparently the PCA computation wasn't stalling out because of the absurd dimensions being reported - updating to tensorflow 1.12 and Keras 2.2.4 has eliminated that problem.

Second, it seems that Keras bases the generation of embedding data for tensorboard on the layer inputs rather than the actual layer weights. Maybe this is important for some other usage, but it makes things very complicated in a standard word/token embedding scenario. To work around this, I ended up skipping embeddings during training, exporting the embedding weights at the end, and then training a new throwaway model with a much simpler topology to generate the projections I'm interested in. Given an object model trained from the samples above:

w <- model %>% get_layer(index = 1) %>% get_weights()

emodel <- model <- keras_model_sequential() %>%
layer_embedding(input_dim = max_features, output_dim = 128, input_length = 1) %>%
layer_global_max_pooling_1d() %>%
layer_dense(units = 1)

emodel %>% get_layer(index = 1) %>% set_weights(w) %>% freeze_weights()

emodel %>% compile(
optimizer = "rmsprop",
loss = "binary_crossentropy",
metrics = c("acc")
)

callbacks = list(
callback_tensorboard(
log_dir = "my_log_dir",
embeddings_freq = 1,
embeddings_data = c(1:max_features)
)
)

history <- emodel %>% fit(
1:max_features, 1:max_features,
epochs = 1,
batch_size = 256,
callbacks = callbacks
)

tensorboard("my_log_dir")


Ideally there would be some simpler/cleaner way to achieve this, but this works for now.

edit

Another user asked in a now-deleted question about adding token labels to Projector. For posterity, that is as simple as adding a value for embeddings_metadata in the callback_tensorboard. This should be a path (relative to your log_dir) to a tab delimited file with the desired labels. Details are here: https://www.tensorflow.org/guide/embedding#metadata