2
$\begingroup$

I am trying to build a ML model that. will take a list of words and will try to produce sentences with those words, based on a language model on an existing corpus. Example:

(freenode, today, neural networks) -> (Today I joined the freenode channel for neural networks....) 

What will be the right (and easiest) approach to do this?

I was trying to find an LSTM RNN based solution, but things get trickier once I feed more than one word to the network.

$\endgroup$
2
  • $\begingroup$ There is a similar recent question with an answer. Just curious, is it a homework problem? $\endgroup$
    – Valentas
    Apr 7, 2022 at 14:20
  • $\begingroup$ @Valentas it is for a pet project $\endgroup$
    – cristid9
    Apr 7, 2022 at 23:17

1 Answer 1

1
$\begingroup$

Some ideas are:

a) Generate many many random sentences until you find your words (will be extremely slow, but the samples will be unbiased).

b) For very short sentences you can iterate over all positions and all possibilities to fill unknown positions like (today, neural networks,?), (?, today, neural networks), (neural networks, ?, today), etc. and use the language model to compute the probability of each sentence, then select the one with highest probability or one at random with weights equal to probabilities. Actually if you can generate all possible positions then you should be able to use a pretrained BERT model directly since it is trained on a similar Masked Language Model task.

c) Train a new language model from scratch, or fine tune a pretrained model (like BERT), with inputs (or sentence pairs) like

today, freenode, neural networks # Today I joined the freenode channel for neural networks.

You can sample the words or n-grams from existing sentences. This might require a large corpus, some work and some cost.

d) If you have access, try existing large models like gpt-3. For example, I tried transformer.huggingface.co, with the last sentence autocompleted by the transformer:

Please make a sentence containing the given words.

Words: horse, tree, walked.

Sentence: The horse walked around the tree.

Words: freenode, today, neural network.

Sentence: The freenode sat on the tree.

Update

Same experiment on https://6b.eleuther.ai/

Sentence: The freenode today uses a neural network to classify images.

$\endgroup$
3
  • $\begingroup$ Can you share the transformer.huggingface.co snippet? $\endgroup$
    – cristid9
    Apr 9, 2022 at 15:48
  • $\begingroup$ I just used their website with the prompt I quoted. You could try gpt-3 beta.openai.com/overview for better results. $\endgroup$
    – Valentas
    Apr 11, 2022 at 12:39
  • $\begingroup$ Ah, right, I didn't pay close attention to it; It seems that it used only the freenodeword. $\endgroup$
    – cristid9
    Apr 11, 2022 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.