1
$\begingroup$

I am new to deep learning. I have a collection of website texts and the activity class each company belongs to. A company can list multiple activity classes (NACE-codes)

The goal is to build a recommender that would recommend the five best activity classes based on a website the user provides.

The model is trained on the top 10K most frequent words that occur in the websites. The words are represented as scores ranging from 0 to 1. Each text is a line in the matrix, with 3000 columns (the first 3K words per website are taken into account).

The labels are k-hot encoded (activity class labels are columns and have 1 if the company lists the activity class; otherwise 0)

There is a significant imbalance between the labels, so each label is weighted by dividing the total number of classes by the time each class occurs in the dataset.

When I train the model, the loss goes down and my custom 'precision' score (correct activity class is among the top-5 predictions) goes up somewhat during training. However, when I use the model to predict, it will always return the same scores between 0 and 1 (what you would expect, given the sigmoid layer) for the classes, no matter what the input is. So, it seems that all the model has learned is that some classes are more likely to occur than others.

Any suggestions or guidelines would be appreciated. I have read Chollet's "Deep learning with python", but it is not exactly a great source of advice on this particular topic.

My model looks like this:

 library(keras)
 metric_precision <- function(y_true, y_pred) {
   metric_top_k_categorical_accuracy(y_true, y_pred, k = 5) 
 }

    model <- keras_model_sequential()
    model %>%
    layer_embedding(input_dim=10000, output_dim=512, 
     input_length=3000, mask_zero = FALSE, trainable=TRUE ) %>%
    layer_dropout(0.3) %>%
    layer_conv_1d(filters,kernel_size, activation = "relu") %>%
    layer_max_pooling_1d(5) %>%
    layer_dropout(0.3) %>%
    layer_conv_1d(filters,kernel_size, activation = "relu") %>%
    layer_global_max_pooling_1d() %>%
    layer_dense(ncol(train_y),"sigmoid")

  model %>% compile(
    loss = 'binary_crossentropy',
    optimizer = 'rmsprop',
    metrics = c(precision=metric_precision)
  )

  model %>% fit(
    train_x, train_y,
    batch_size = batch_size,
    epochs = epochs,
    validation_data = list(test_x, test_y),
    class_weight = classweights
  )
$\endgroup$
  • $\begingroup$ When I encode the words as integers and re-train, the model provides different probability estimates based on the input $\endgroup$ – Jeroen Claes Mar 3 '18 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.