Abstract:
Late in his career, Daniel Dennett changed his mind about minds. If his new view is correct, it has deep implications for both the moral standing of nascent AIs and for the existential danger they may pose. He suggests in his (2017) that genuine minds require a different kind of architecture than he previously thought: one built from parts that themselves have simpler forms of caring and agency. My presentation builds on this suggestion—and the work by Terrence Deacon (2013) that inspired it—to develop a kind of “desire homuncularism”, according to which complex desires with reference to the world must be built out of simpler desires. Systems that intuitively don’t “really care”, and have no “skin in the game”—systems like DeepMind’s AlphaGo, and perhaps like modern LLMs—seem to lack this feature.
The stakes for getting these questions right are high. First, the degree to which nascent AIs possess “real caring” bears directly on what moral status, if any, we should accord them. A system with real cares, after all, plausibly thereby has interests and a proto-welfare. Second, which AI systems “really care” connects to questions of artificial agency and the existential risk from unaligned superintelligent systems. An AI system indifferent to its goals seems more like a tool, more amenable to being safely shut down; one with genuine skin in the game, in contrast, is more likely to fight for its goals, and pose a more serious existential threat (Omohundro 2008).
Dennett suggests the kind of “real caring” at issue also seems closely connected to what systems should be attributed minds. I claim this fits with the increasingly prominent naturalistic theory of mental representation known as teleosemantics. On this view, a system’s representational contents depend on some kind of teleology—one that must be more than the mere functional capacities of a hammer or a thermostat. Furthermore, the puzzling tendency to attribute minds to the very same systems to which we attribute moral status can be parsimoniously explained if these two properties share a common foundation in “real” caring.
The main focus of the paper, however, is on the more radical claim that genuine caring requires component parts that themselves care. Here Dennett points to arguments in Deacon’s enormous and ambitious book that are understandably difficult to summarize. My paper attempts to construct a fuller case for a view in the same spirit.
Deacon suggests that genuine teleology emerges, at the base level, from particular patterns of constraint. In the simplest case, two dissipative far-from-equilibrium processes can mutually interfere with each other’s entropy production rate, thereby reciprocally contributing to the other’s prolonged existence. From such simple beginnings, Deacon argues, nature can build ever more elaborate levels of teleology. Crucially, these teleological systems maintain themselves through active work by their parts, rather than through passive resistance to change like typical mechanical systems.
Drawing on this framework, I develop two key arguments for desire homuncu- larism. First, systems with distributed caring exhibit deeper integration, as evidenced by how changes to their parts affect their overall goals. As an il- lustration, I contrast stereotypical corporations with volunteer clubs. When a corporation replaces an employee, its fundamental goal of profit maximization remains unchanged. But when a club gains or loses members, its goals shift because they emerge from the Pareto frontier of members’ diverse motivations. Similarly, I argue that genuine minds require goals that are constituted by, rather than merely served by, their parts. Note that independent issues in philosophy of mind also need a clearer notion of integration:
• Descartes’ observation that we are not related to our bodies as sailors in ships, who are informed only intellectually of damage; instead we are “intermingled” with our bodies (Descartes 1641)
• The suggestion in embodied approaches to emotion that we must “listen down” to our bodies to discover our values (Damasio 1994)
• The search for a principled line between the mental and the environment in debates over the “extended mind” (Clark and Chalmers 1998; Sprevak 2009)
Second, I claim systems with such distributed goals can achieve a richer form of conative reference—that is, their goals are more firmly grounded in the world, and harder to “spoof” or “wirehead”. Just as Dennett argued that a sophisticated thermostat with multiple sensory channels would achieve richer indicative reference than a simple bimetallic strip, systems with distributed caring achieve richer imperative reference than systems with centralized goals. A traditional corporation’s singular profit motive could be realized by a wide variety of states of the world, and potentially spoofed by manipulating spreadsheet numbers. Spoofing a volunteer club’s distributed goals, on the other hand, requires spoofing each member’s diverse motivations.
Together, these claims help explain why we intuitively deny that systems like AlphaGo truly care about winning—though their models of the game are com- putationally complex, their goals remain referentially “thin”, like a traditional thermostat’s. It may be the same for modern large language models; again, though these systems possess remarkably complex models of the world, their goals arguably remain as referentially thin as AlphaGo’s. They seem to be, in Keith Frankish’s apt phrase, “cognitively rich but conatively bankrupt.” Even as such systems become more embodied and multi-modal, as with Google’s PaLM- E, they may lack the kind of distributed, bottom-up caring that characterizes genuine minds with moral standing.
Thus—to my surprise and chagrin—desire homuncularism suggests there may be kernels of truth behind naive intuitions that AIs cannot have minds because they are “just programmed” and “not alive”. Minds may require the kind of bottom-up, distributed caring characteristic of living systems. However, this requirement is functionally specified; nothing in principle prevents non-biological systems from instantiating the relevant patterns of constraint and teleology.
Important questions remain, such as whether this view implies moral status for highly integrated collective entities like clubs or nations. But at minimum, desire homuncularism offers a novel framework for thinking about minds, moral status, and artificial intelligence, while developing an intriguing late-career shift in Dennett’s influential views on mind and consciousness.
