Metric to evaluate words generated by Neural Network

Question

I have this task at hand and I would be grateful for some directions. Perfectly not the final solution as I would like to do it myself.

Let's say I need to create new fruit names based on existing ones.

I made it using textgenrnn library.

Now I have a list of potential new names. I would like to build a metric which would calculate that e.g.:

1. (Fabic, Alis, Brooty, Morange etc) are potentially good fruit name
2. (Ssae, Sriew, Adeoie, Seeea) are potential bad fruit names.

Is there any list of metrics for word generating?

As of now, I found following resources:

• Evaluating Generative Models for Text Generation https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1174/reports/2737434.pdf

• Character-Level Language Modeling with Hierarchical Recurrent Neural Networks http://150.162.46.34:8080/icassp2017/pdfs/0005720.pdf

• Graves A. Generating Sequences With Recurrent Neural Networks https://arxiv.org/pdf/1308.0850.pdf

• Subword language modeling with neural networks. http://www.fit.vutbr.cz/~imikolov/rnnlm/char.pdf

But since I learnt about them today, I would need a more vanilla introduction to this topic of validation.

Answer 1

Maybe somebody will have a better idea but the default method would be to generate a set of names, then ask a few annotators to label them as good or bad (possibly scoring them from 1 very bad to 5 very good), and finally train a supervised model to recognize the good from the bad ones. This approach would also give you the opportunity to check the inter-annotator agreement, i.e. assess how subjective the choice good vs. bad is. In general a subjective task is difficult (or impossible) to do with an unsupervised metric.