Can AI truly “think” or “know” something? Does ChatGPT understand language or is it just a stochastic parrot? (1 minute read)
Searle’s famous Chinese Room argument likens a computer to an English speaker sitting in a room, equipped with instructions for responding to Chinese prompts. Searle argues that, although from the outside the person in the room may appear to understand Chinese and could pass the Chinese Turing Test, they are merely executing instructions. He concludes that a computer cannot truly think or understand language.
Personally, I agree there is no genuine understanding in Searle’s scenario. Yet, I think that it is a misleading metaphor for modern neural-network-based AI, such as ChatGPT-4.
Such networks are more akin to the human brain than to the if-this-then-that instructions found in good ol’ AI or these followed by the Chinese room operator. Chinese Room—along with its intuitive interpretations—remains useful, but only applies at the level of individual neurons, which process their input following a set of instructions and pass the results on to other neurons. Such Chinese-Room-like neurons do not think nor understand. Yet, networks of neurons do.
Consider the brain of a native Chinese speaker. We would readily credit it with consciousness or understanding of Chinese—even though none of its neurons is conscious or understands. Now, imagine a flawlessly functional replica of this brain, where the neurons are replaced with neuron-shaped versions of Searle’s Chinese rooms. Each room contains instructions and machinery that allow its microscopic operator to flawlessly emulate the behavior of the original neuron, from undergoing action potentials to releasing neurotransmitters. Such a replica should be credited with the properties of the original brain such as understanding or consciousness. This holds even though, in accordance with Searle’s argument, the operators of the neuron-rooms do not comprehend Chinese and they would find it unlikely that their collective activity could generate something of that nature.
In other words, networks of neurons manifest emergent properties that are not present in individual neurons and cannot be anticipated or deduced by studying them in isolation.
But what about smaller neural networks, such as ChatGPT? Does it understand language? Let’s run a thought experiment again.
Take a single Chinese-Room-like neuron. Then, progressively add neurons, arranging them into a multilayered network akin to one employed by ChatGPT. Once you incorporate a few million neurons, train your network to predict the subsequent words in Chinese texts. Room operators begin with random instructions, but they refine them based on feedback from their recipients. Given enough training cycles&data and you will get the Chinese version of ChatGPT, capable of generating language indistinguishable from that produced by humans.
Next, outfit the rooms with additional machinery, such as neurotransmitter pumps, and continue expanding and reconfiguring the network until you obtain the perfect replica of the Chinese speaker’s brain. Such a replica should be credited with thought and understanding.
At which stage in this evolution—from a single neuron resembling a Chinese Room, through a few million neurons capable of generating language, to a perfect brain replica—should we credit the network with human-like mental capacities such as “understanding”? It seems preposterous to claim that mental capacities should be credited to a single neuron, even if it could tackle some language tasks. It seems equally unreasonable to argue that a brain replica should lose its mental capacities the moment we remove some neurons or restrict their functionality. (Human brains can maintain much of their potential despite significant loss of neural mass and function caused by aging or disease.)
In essence, “understanding” must emerge somewhere between a single neuron and a full brain replica. Does it occur before, alongside, or after the neural network gains the ability to handle language at human level? What do you “think”?
More here: arxiv.org/abs/2302.02083