I am working on a text-classification problem, and trying to understand how to work with the tensorboard projector for an embeddings layer in Keras. Borrowing an example from the Deep Learning with R book, I have a model set up like this:
library(keras) max_features <- 2000 max_len <- 500 imdb <- dataset_imdb(num_words = max_features) c(c(x_train, y_train), c(x_test, y_test)) %<-% imdb x_train <- pad_sequences(x_train, maxlen = max_len) x_test = pad_sequences(x_test, maxlen = max_len) model <- keras_model_sequential() %>% layer_embedding(input_dim = max_features + 1, output_dim = 128, input_length = max_len, name = "embed") %>% layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>% layer_max_pooling_1d(pool_size = 5) %>% layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>% layer_global_max_pooling_1d() %>% layer_dense(units = 1) summary(model) model %>% compile( optimizer = "rmsprop", loss = "binary_crossentropy", metrics = c("acc") )
When it comes time to configure the
callback_tensorboard object, however, I'm a little lost. It seems the API has changed since the book was written, and I haven't found a good working example yet. The
embeddings_data property is apparently required if the
embeddings_freq parameter is set, and it needs to match the shape of the model inputs
c(?, 500). I can satisfy this by simply passing all the word tokens as a matrix:
tensorboard("my_log_dir") callbacks = list( callback_tensorboard( log_dir = "my_log_dir", histogram_freq = 1, embeddings_freq = 1, embeddings_data = matrix(1:max_features, nrow = 4) ) ) history <- model %>% fit( x_train, y_train, epochs = 5, batch_size = 128, validation_split = 0.2, callbacks = callbacks )
The model trains OK, but when trying to look at the result in projector, it seems that the embeddings are being processed as 64,000 dimensions (2,000 words * 128d layer output?) with a single point for each 500 element vector in
embeddings_data. This makes PCA computation stall out.
I expect a final result closer to the example at https://projector.tensorflow.org/, where Dimensions are equal to the layer output dimensions, and there is a single point for each word.
What am I missing? Are there any good working examples of visualizing a standard word embedding layer with current versions of Keras and Tensorflow?
I am using
Keras 2.2.0, and