Evaluating Language Model on specific topic

Question

I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in the form of either a single keyword(e.g. 'Computers') or a bag-of-words(e.g. 'Computers', 'Linux', 'Server'...).

For example given a LM, how relative are the outputs of the model to the topic specified by the word Computers?

What I have already tried: Generating a large enough number of sentences from the LM and taking the average cosine similarity between these sentences and the target topic(or every word in that topic we have more than one) as described here . I am not sure if this is a valid way to go and furthermore the cosine similarity between sentences yields poor results in many cases.

Thanks in advance for any help.

Answer 1

I think there are (at least) two parts to take into account in evaluating such a model:

• Whether the generated text correctly relate to the input topic
• Whether the generated text is grammatically and semantically acceptable

In my opinion the first kind of evaluation could reasonably be done with an automatic method such as the one you propose. Note that cosine scores should not be interpreted absolutely: you should probably compute cosine similarity with a random sample of topics, and normally one expects the similarity to be much higher with the input topic than any other. You could also think of other variants, for instance training topic models on the generated text together with a sample of documents from various known topics, then check that the generated text belongs to the target topic (i.e. it should be grouped with the documents known to belong to this topic).

For the second kind of evaluation, it would be difficult and unreliable to use an automatic method. As far as I know the only reliable way would be to ask human annotators to assess whether the text is grammatically correct and whether its content makes sense. If you're going to do that you might as well ask them to annotate to what extent the text is related to the topic.

---

[added following comment]

if you check whether the generated text is similar to the topic only by computing similarity with this target topic, what you obtain is for instance an average cosine score. Then you would probably select a threshold: for instance if the similarity is higher than 0.5 then consider that the text is indeed related to the topic. But there are two problems with this option:

• In some cases the average similarity will be lower than the threshold even though the text is correctly related to the topic. This could happen for example with a very "broad" topic which covers a large vocabulary.
• On the contrary you might have cases where the average similarity is higher than the threshold, but actually comparing to another topic would give an even higher similarity value.

These issues are due to interpreting the similarity score "absolutely", as opposed to interpreting it relatively to other similarity scores. Instead you can calculate the similarity not only against the target topic but also against other topics, and then just check that the target topic is the most similar topic (or at least one of the top similar). This way :

• The target similarity score may be low, as long as it's higher than the other topics
• you can detect the case where another topic happens to have higher similarity than the target topic

Answer 2

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that.

However, you should take into account that it is possible that your model generates very similar sentences always. For that reason, people normally evaluate not only quality but also diversity. For this, you can compute the "self BLEU" of your generated sentences, that is, you compute the BLEU of each generated sentence against the rest of the generated sentences. You can find a script for that here.

You should also take into account that you can trade quality for diversity and vice versa, by setting different values of the output softmax temperature. For this, you may want to evaluate your model at different values of the temperature, to understand different generation regimes, plotting them as a curve, like this:

You can find more about this kind of evaluation in this article, where I took the figure from.

You can find alternatives to the quality-diversity evaluation here.