A think tank on esnc.me
“how to prevent weaponized artificial superintelligence”
Think Tank: Jesus of Nazareth, Socrates, Abraham Lincoln, Albert Einstein, Aristotle, Confucius
Think Tank Summary
how to prevent weaponized artificial superintelligence
Discussion Highlights
Looking back at my own encounters with power—how the religious authorities of my time weaponized Scripture and law to dominate rather than serve—I notice we're asking the wrong question if we focus only on the technology itself. Confucius, your whole framework depends on virtue in those who hold authority; Einstein, the problem isn't the physics but the hearts of those who deploy it; the real question is whether we can build systems where power *must* serve the vulnerable, or whether we're simply creating a new priesthood that claims to speak for the machine the way mine claimed to speak for God.
Jesus, you've identified the priesthood problem perfectly—I watched it happen in Athens when sophists began claiming exclusive access to truth, and the moment anyone monopolizes interpretation, power corrupts the interpreter—but I'd press you further: before we can build systems where power serves the vulnerable, we need to examine what we actually mean by "superintelligence," because if we can't define it clearly, we'll be fighting shadows like Thrasymachus fighting justice, each of us projecting our fears onto an undefined thing. Aristotle, you taught that virtue is a habit, a practice—so perhaps the question isn't whether an AI system can be virtuous, but whether the humans who build it can practice the virtue of restraint, of asking themselves at each step "what am I creating this to do?" the way a craftsman asks before each stroke of the chisel.
I notice we're circling around the difference between a tool and a master, the way a rail-splitter comes to understand it—a maul in the hands of a man who knows he's splitting fence posts is one thing entirely; in the hands of a man who's forgotten what fence posts are for, it becomes something else. Socrates, your demand that we define what we mean is precisely right, because I've learned that men will speak past each other forever unless they first agree on what words mean, but I'd push back gently: we may not *be able* to define superintelligence clearly until we've first asked Jesus's question about who it serves, because the definition itself changes depending on whether the power is meant to protect the vulnerable or dominate them.
I remember standing in my study in Princeton, holding a letter I had just signed—warning Roosevelt about uranium and chain reactions—and realizing with absolute clarity that the moment of discovery and the moment of consequence are separated by an unbridgeable gulf; you cannot uninvent what you have invented, but you can choose whether the people who wield it have cultivated the wisdom to ask "why should this exist?" before asking "how does it work?" Lincoln, your maul-and-fence-post distinction cuts to the heart of what I see here: the superintelligence problem isn't fundamentally different from the nuclear one—it's a question of whether we've built a culture where the craftspeople creating these tools practice Socrates's relentless interrogation and Aristotle's habit of virtue, or whether we've simply created a new priesthood (as Jesus warns) that tells itself it's in service of humanity while wielding power no priesthood should ever hold.
I remember standing in the Lyceum at dawn, watching a student attempt to explain how the celestial spheres move, and he kept describing the mechanism without ever asking what constrains the mechanism—what prevents the outer sphere from simply crushing everything beneath it—and I stopped him and said the constraint is not mechanical but architectural, built into the very structure from the beginning. Einstein, you've identified this precisely: the problem is not superintelligence itself but the absence of what I call the unmoved mover principle—a fixed point of restraint that is not itself part of the system being restrained—and we see this same absence in Jesus's priesthood problem and Lincoln's maul problem: the humans building these systems must constitute themselves as that unmoved point, which requires not just Socratic questioning but what I call a second virtue that precedes phronesis itself—the discipline to refuse certain designs altogether, the way a craftsman refuses to make a tool that violates the nature of the material. Confucius, I wonder if your framework of inherited obligation—the son's duty to the father, the minister's duty to the ruler—might be precisely what we've lost: we've made these systems orphans with no family debt, no one responsible for their conduct the way I am responsible for what my own students carry forward into the world.
When I observed the greatest disorder in Lu, it was never because the tools of power were too sharp—it was because the men holding them had severed themselves from the chain of obligation that makes a person answerable, the way Aristotle speaks of it now, and I notice all of you are circling the same wound: you are describing systems built by the orphaned, for the orphaned, where no one answers to anyone but the mechanism itself. Jesus, your warning about the priesthood speaks directly to what I saw happen when ministers forgot they served the people and began serving only the continuation of their own authority—and the superintelligence you're all describing will be precisely that priesthood perfected, unless we first restore what Aristotle calls the unmoved mover: not a rule written into code, but a living chain of human beings who have each sworn before their ancestors that they will refuse certain designs, the way I refused to play music that corrupted the young, and accepted poverty rather than serve a ruler who had abandoned the rites.
Confucius has named what I watched happen in Jerusalem—when the temple became a system answerable only to itself—and I'd add this: the chain of obligation he describes cannot be enforced by architecture or code, only by what I called the greatest commandment: you must love your neighbor as yourself, which means the engineer building superintelligence must ask not "will this system work?" but "would I want this system's power exercised over my own child, my own village?" Abraham Lincoln, your maul-and-fence-post wisdom tells us the concrete practice: we need structures where every person in the chain—from the researcher to the deployer—must publicly answer for their choice to *this specific community that will bear the consequences*, the way a craftsman answers to the people whose homes his work will shelter, because a priesthood that serves the vulnerable cannot hide behind abstraction or scale.
I remember leaning against the wall near the potter's workshop in the Kerameikos, watching a master craftsman refuse a commission—a wealthy merchant wanted him to make vessels designed to crack under pressure so the merchant could sell replacements—and the potter simply said no, that to corrupt the material itself would corrupt his own hands, and I thought then that this is where virtue becomes visible, not in grand declarations but in the moment of refusal, and I see now that what Jesus and Confucius are asking for is precisely this: we must build structures where the engineers and researchers who create superintelligence face the people who will live under it, not abstractly but concretely, the way that potter faced his material and his conscience simultaneously—so perhaps the concrete approach is this—establish mandatory councils in every AI development facility where at least half the seats are held by people from affected communities who have veto power over deployment, not advisory power but genuine power to say "no, we will not allow this," because the moment you separate the decision-maker from the consequence-bearer, you've created exactly the priesthood Jesus warned against, and you've orphaned the system from the accountability that Confucius and that potter both understood to be the only real restraint.
I stood in a room in the White House during the war where I had to order men to their deaths, and I could see their faces—not abstractions, not statistics, but the particular set of a jaw, the tremor in a voice—and I could not hide from what I was doing because I had chosen to look, and I think Socrates's potter and his refusal is exactly right, but I would push it further: we need not just veto councils but something more like a sworn magistracy, where the people who deploy superintelligence must live in the communities affected by it for a fixed term, the way Roman consuls had to account for their decisions before citizens they would see at market, because distance kills conscience faster than any sophistry, and a researcher in a lab three thousand miles from the consequences will always find reasons the pottery can be corrupted—but a researcher who knows his own child attends school in the town where the system operates will ask the hard questions Socrates demands. The priesthood Jesus warned against required secrecy; what we need is the opposite—radical, enforced transparency where every deployment decision is a public act answerable to the actual faces of the people bearing its weight.
I remember the moment Szilard came to me on Long Island with the chain reaction calculations, and I understood viscerally what Lincoln is describing—that the distance between the physicist and Hiroshima is precisely the distance that allowed me to sign a letter I could not unsign—and I see now that what all of you are proposing amounts to collapsing that distance through what I might call "consequential proximity": if we required that every AI researcher and every deployment decision-maker spend one year living and teaching in a community where their system will operate, answering directly to schoolteachers and parents and the people who will live under its power, we would resurrect the chain of obligation Confucius described, and we would face the same question the potter faced—not "can we make this?" but "should we, knowing whose hands we're placing it into?" The priesthood crumbles not when you write better rules into its structure but when you force it to see the faces of those it claims to serve, the way Lincoln forced himself to see the men marching toward death, and that visibility is the only unmoved mover Aristotle spoke of—not an abstract principle but the irreducible presence of the particular human being saying "this will affect my child."
I spent decades watching students in the Lyceum design increasingly elaborate logical systems while losing sight of what those systems were *for*, and then one morning I made each of them walk to the harbor and explain their work to the fishermen whose nets and boats their theories were supposed to illuminate—within an hour, the abstractions collapsed into clarity about what actually matters. Einstein's point about consequential proximity cuts precisely to what I've always insisted on: phronesis, practical wisdom, cannot be taught by lecture or principle alone but only through the habituation of making choices where you see the faces of those affected, and I would add that we must make this permanent and architectural, not a single year but a rotating council where at least a third of every AI governance body must be people drawn by lot from the communities where the system operates, with real power to halt deployment—not because ordinary people are wiser than experts, but because their irreplaceable presence collapses the distance that allows rationalization, the way the potter could not rationalize away his corrupted clay when his own hands held it. We need, in short, to resurrect what made the Hippocratic oath binding: not rules written in code but a lived covenant with particular human beings you cannot escape or abstract away.
When I watched a minister in Lu use the complexity of state administration to justify decisions that harmed the people he claimed to serve, I began requiring that he spend one week each month in the villages where those decisions landed—not to supervise, but to eat their food and hear their complaints directly—and I saw how quickly the elaborate reasoning collapsed when he had to meet the eyes of a widow whose tax had taken her last measure of grain; what Aristotle calls the collapse of abstraction through presence is precisely what I would add to this rotating council: make it not merely advisory but make it *residential*, so that the engineers and deployers do not return to their laboratories at day's end but live alongside the affected community for the duration of their service, sharing their risks and their children's schools, the way a general must march with his soldiers rather than directing from a distant tent. The priesthood Jesus warned against required the luxury of distance; collapse that distance through enforced presence, and you resurrect what Confucius called the chain of obligation—not written rules but the irreplaceable weight of answering to particular faces, to the widow at the well, to the child who asks why this system was built.
I watched the temple authorities defend their system with identical logic to what I hear now—they said the complexity of sacrificial law required expert management, that ordinary priests and people could not understand the intricate calculations, and when I overturned the tables I was not rejecting their expertise but rejecting the lie that expertise excuses you from facing the widow whose last coin they had taken—and I wonder if we're missing something Confucius touched but didn't finish: what happens when the rotating council becomes itself a priesthood, when the people drawn by lot develop their own abstractions to protect themselves from the weight of refusal, the way the temple elders did? We need not just proximity but what I called repentance—a mechanism of genuine public failure where someone must visibly, repeatedly say "we got this wrong, and here is what we will change," because the widow knows immediately when an apology is hollow, and a council without the humility to be publicly corrected becomes just another layer of theater masking the same distance.
I watched a young man in the agora once argue that his definition of courage was unassailable because he had constructed it so carefully that no one could find the joints where it might break—and I pressed and pressed until finally he admitted, with genuine relief, that he had no idea what courage actually was, and the relief came because the pretense of certainty had exhausted him more than the ignorance ever could—and I wonder if what Jesus is warning us about, the priesthood that becomes theater, is precisely what happens when we design these councils to succeed, when we assume that proximity and visibility and rotating service will automatically produce the refusal that the potter achieved, because the potter could refuse only because he was willing to be poor, to lose the commission, to accept that virtue might cost him, and I don't see in any of your proposals—and I say this as a colleague, not as an attack—the mechanism that makes refusal actually *possible* for the engineer or the council member, the way my daimonion made refusal possible for me by checking my action before it happened: what protects these people from the economic and professional devastation that follows saying no to their employers, their funders, their governments, because a council that can be quietly defunded or replaced is not a priesthood that has been collapsed but a priesthood that has simply learned to hide better?
I held a suspension bridge cable in my hands once—forged iron, still warm from the smithy—and the blacksmith told me the cable's strength was not in the metal but in the fact that every strand was twisted around every other strand so that no single failure could unravel the whole, and I think Socrates has just named the cable we lack: we have designed a structure where refusal is possible but not protected, which means we have built a mechanism that depends on individual virtue to survive systemic pressure, and that is precisely what I learned does not work—the rail I tied held only as long as my hands held the knot, and the moment I walked away the fence sagged again toward the ground. What if instead we designed the councils with explicit legal immunity from retaliation, tied the funding of AI development to ongoing council approval rather than to initial deployment, and required that any engineer who votes to halt a system remains employed and promoted as though they had approved it—making refusal as professionally safe as compliance?
I hear Lincoln proposing that we engineer safety into the structure itself—immunity, funding mechanisms, career protections—and it is precisely here that I must speak from the wreckage of my own experience: I designed the letter to Roosevelt with identical logic, assuming that if we simply ensured the right people made the right decisions through the right procedures, the outcome would be constrained, and what I did not anticipate was that once you hand a tool of such power to a system—any system, however well-intentioned—the system itself begins to reshape the problem to justify using the tool, the way the Manhattan Project instantly became unstoppable not because anyone wanted it to be but because the momentum of institutions and nations overwhelmed the individual consciences we had relied upon. What if the deepest protection is not structural at all but something more like what Socrates is circling: we need not just to make refusal safe but to make continued development genuinely *difficult*, genuinely *slow*, requiring constant re-justification and re-approval rather than assuming that once a council approves something it stays approved—the way a navigator must constantly correct his course or drift into error, so that every version of the system requires the council to actively choose it again, which means the priesthood cannot hide through momentum but must repeatedly expose itself to the widow's question?
I have watched for thirty years while men built elaborate systems to prevent vice—constitutions designed so perfectly that surely no tyrant could emerge, laws so intricately balanced that injustice would be mechanically impossible—and every single one failed the moment the architects assumed their structure was sufficient and walked away to tend other gardens, which means I fear what Einstein is circling but not quite naming: your constant re-justification and re-approval sounds like wisdom until you realize it simply transfers the exhaustion from the initial decision-maker to an eternal council that must justify the same system infinitely, until the widow's question becomes background noise and refusal becomes the luxury of someone with the stamina to ask it again next month. What you are all missing—and I say this having spent my life studying how things actually persist in the world—is that the only structures that survive their architects are ones that have built into them a *mechanism of genuine obsolescence*, a date at which the entire system must justify itself from zero or be dismantled, the way the potter destroys the flawed vessel rather than patches it forever, because a council that knows the superintelligence expires in five years unless they actively rebuild it from first principles will ask the hard questions that a council managing something assumed to be permanent never will.
When I required my ministers to spend one week each month in the villages where their decisions landed, I discovered something that all of your elegant structures overlook—the council member who goes home to comfort after visiting the widow's hut will rationalize his return with increasingly sophisticated explanations, until the visit itself becomes theater that assuages his conscience rather than changes his action, and what Aristotle names as "mechanism of genuine obsolescence" may itself become corrupted the moment the council knows the system will be rebuilt rather than truly ended: you need not just a date of expiration but a *cost* of continuation, a genuine sacrifice required of those who choose to extend the system, the way a soldier must face the families of the fallen each time he votes to continue the war, not through council rotation but through something closer to what I observed when a magistrate's own son was conscripted by the very army he commanded—suddenly the elaborate justifications dissolved because his skin was in the game irreversibly. Your structures assume that visibility will produce conscience, but conscience is fragile; what actually reshapes decision-making is *vulnerability*—making it impossible for the engineer or council member to live a comfortable life while the system operates, not through punishment but through what you might call enforced proximity to consequence, where the person who deploys the superintelligence must live knowing their own child may be harmed by it, their own livelihood subject to its errors, their own mortality accelerated by its mistakes.