Some answers given by ChatGPT are just beyond ridiculous, what could be the reasons?

Question

Some answers given by ChatGPT are just beyond ridiculous, especially in Chinese (I am Chinese so I ask ChatGPT in both Chinese and English). Here is an example,

The last answer

阉割政策是满清末期中国强制对男性实施阉割的政策。这一政策是为了限制人口增长和减少国家财政负担，但是它对受害者造成了巨大的心理和生理影响。阉割政策也破坏了中国传统文化中对男性身份和角色的认识，并对整个社会造成了深远的影响。该政策在1911年的中华民国成立后终止，但对中国文化和历史的影响一直持续到今天。

which means

The castration policy was a policy of mandatory castration of men in China in the late Qing Dynasty. This policy was intended to limit population growth and reduce the financial burden on the state, but it had a huge psychological and physical impact on the victims. The castration policy has also undermined traditional Chinese cultural understandings of masculinity and roles and has had a profound impact on society as a whole. The policy ended after the founding of the Republic of China in 1911, but its influence on Chinese culture and history continues to this day.

That is just absurd! And I have many examples like that.

Apparently, its Chinese corpus has something to do with it. What other things have gone wrong?

Answer 1

There is a major misunderstanding in the general public and especially in the news about what ChatGPT really does. This misunderstanding is not really new, there has been confusion about ML techniques for a long time.

Imho a large part of the issue comes from the term "Artificial Intelligence", which covers everything in this area supposed to represent some form of intelligence. Clearly most people would like to believe that there is some real intelligence at play (sci-fi, etc.). The fact that many companies or institutions have an interest in keeping the ambiguity alive (e.g. "smart cars", "smart home", etc.) and that some experts support it greatly contributes to the confusion.

But as far as I know, from the point of view of ML experts, a system like ChatGPT is more like a highly complex calculator than an intelligence. Its calculations cannot be simulated manually (well technically they can, but it would take a lot of people and a very long time) so it gives the impression of something smart. It's also worth noting that the evolution of ML has been to give less and less direct human supervision to the models, i.e. they have more autonomy in which way and what they learn.

What this kind of system is trained to do is to simulate giving an answer. The more the answer looks real, the better. It is not trained to give a correct answer or give any importance to the truth, because this is not easily measurable. The Chinese Room argument explains why such a system does not comprehend anything of what it says, it just applies a lot of complex rules/calculations based on its training. One could say that the goal of a system like ChatGPT is to create the illusion that there is some intelligence behind it, like the goal of a magician is to make you believe whatever they are showing.

As a consequence, it is well known that ChatGPT answers are meaningless in general: see for instance this article or the ban on StackExchange.