If I wanted to learn constants for example week -> 7 days, chicken -> 2 legs, day -> 24, 1km -> 1000 meters hours, and so on, would it be possible to extract this information from a BERT model trained on the right dataset like Wikipedia words? If not, what model would I have to use?


Absolutely. You can use this gist by Yuchen Lin to get the correct answers from BERT:

predict_masked_sent("There are [MASK] days in a week.", top_k=1)
> [MASK]: 'seven'  | weights: 0.1132921576499939

predict_masked_sent("A chicken has [MASK] legs.", top_k=1)
> [MASK]: 'four'  | weights: 0.25219154357910156

predict_masked_sent("1 km = [MASK] m", top_k=1)
> [MASK]: '500'  | weights: 0.08255643397569656

predict_masked_sent("1 day = [MASK] hours", top_k=1)
> [MASK]: '24'  | weights: 0.06566877663135529

Following is a direction you might want to explore -

You can build a closed domain question answering model using massive amounts of text. And then give a single word as an input to the model. More details about this can be found here. In short, in closed-book question answering, large language models like T5 or GPT-3 memorize some facts during pre-training and can generate an answer without explicit context (unlike normal question answering models).

You will need to build and annotate the datasets accordingly for your task.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.