Some answers given by ChatGPT are just beyond ridiculous, especially in Chinese (I am Chinese so I ask ChatGPT in both Chinese and English). Here is an example,


The last answer


which means

The castration policy was a policy of mandatory castration of men in China in the late Qing Dynasty. This policy was intended to limit population growth and reduce the financial burden on the state, but it had a huge psychological and physical impact on the victims. The castration policy has also undermined traditional Chinese cultural understandings of masculinity and roles and has had a profound impact on society as a whole. The policy ended after the founding of the Republic of China in 1911, but its influence on Chinese culture and history continues to this day.

That is just absurd! And I have many examples like that.

Apparently, its Chinese corpus has something to do with it. What other things have gone wrong?


1 Answer 1


There is a major misunderstanding in the general public and especially in the news about what ChatGPT really does. This misunderstanding is not really new, there has been confusion about ML techniques for a long time.

Imho a large part of the issue comes from the term "Artificial Intelligence", which covers everything in this area supposed to represent some form of intelligence. Clearly most people would like to believe that there is some real intelligence at play (sci-fi, etc.). The fact that many companies or institutions have an interest in keeping the ambiguity alive (e.g. "smart cars", "smart home", etc.) and that some experts support it greatly contributes to the confusion.

But as far as I know, from the point of view of ML experts, a system like ChatGPT is more like a highly complex calculator than an intelligence. Its calculations cannot be simulated manually (well technically they can, but it would take a lot of people and a very long time) so it gives the impression of something smart. It's also worth noting that the evolution of ML has been to give less and less direct human supervision to the models, i.e. they have more autonomy in which way and what they learn.

What this kind of system is trained to do is to simulate giving an answer. The more the answer looks real, the better. It is not trained to give a correct answer or give any importance to the truth, because this is not easily measurable. The Chinese Room argument explains why such a system does not comprehend anything of what it says, it just applies a lot of complex rules/calculations based on its training. One could say that the goal of a system like ChatGPT is to create the illusion that there is some intelligence behind it, like the goal of a magician is to make you believe whatever they are showing.

As a consequence, it is well known that ChatGPT answers are meaningless in general: see for instance this article or the ban on StackExchange.

  • $\begingroup$ I understand everything you said but I still don't understand why the answer was so wrong, especially compared to its English version. $\endgroup$
    – Qiulang
    Commented Feb 8, 2023 at 12:05
  • $\begingroup$ @Qiulang Very likely there were more and better quality resources used for the training of the English system than the Chinese one. Additionally it's very likely that the English version was tested/tuned/polished more thoroughly. $\endgroup$
    – Erwan
    Commented Feb 8, 2023 at 12:10
  • $\begingroup$ @Qiulang, perhaps it's easier to understand when perceiving that bot as a (complicated) filter: provide input, and it will provide output, perhaps unpredictable one. Said machine/technology doesn't dream, can't tell right from wrong, can't reflect or express ITSELF etc. It's just a stupid filter. // The intelligent part, sat, still sits and will sit on the other side of the keyboard ... and we are so willing to give into the illusion of "intelligent behaviour". That's all there is. $\endgroup$
    – MS-SPO
    Commented Feb 8, 2023 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.