I won't go into the meaning of "knowing" here.
But could you teach an LLM something while also teaching it to generally never divulge that knowledge? How would you test this knowledge?
A hypothetical example would be an unsafe chemical reaction. The model should never tell a user how to make an unsafe thing. But it should "know" about this reaction , its uses (legit and illicit ), how it dangerous and also how to create safe environment in which to do it. But if a seemingly unaware user says "hey i am going to mix these together" it should in this case warn them. For qualified knowledgeable users it would be useful for the LLM to know that this information as two reactions that are normally safe could have byproducts that when combined lead to the unsafe reaction.
To be clear I'm not chemical engineer and i am using a chemical reaction as example here and not proposing people take chemistry advice from an LLM.
Another example would be knowing how a particular type of scam generally works so the model can warn a user something sounds like scam. But should never tell a user how to setup or create scam.
