I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in the form of either a single keyword(e.g. 'Computers') or a bag-of-words(e.g. 'Computers', 'Linux', 'Server'...).
For example given a LM, how relative are the outputs of the model to the topic specified by the word Computers?
What I have already tried: Generating a large enough number of sentences from the LM and taking the average cosine similarity between these sentences and the target topic(or every word in that topic we have more than one) as described here . I am not sure if this is a valid way to go and furthermore the cosine similarity between sentences yields poor results in many cases.
Thanks in advance for any help.