"Who Done It?" - Can AI Kill Us?

## Editorial

### "Who Done It?" - Can AI Kill Us?

Jonathan Rauch and I discussed the dangers of AI, invited by Andrew Keen. It was great and Jonathan was a gentleman as he sought to (actually I’m not sure what he hoped for). It was fun and is this week’s video. Jonathan asked me the question directly.

Why are you so confident that AI is benign? It is the right question for this moment, but I did disclose a bias before answering it. Most people react against change instinctively, because it disrupts habit. There is always ambiguity in change. Because of that business models always emerge to monetize fear and doubt. Sometimes that takes the form of headline driven clickbait like this week’s title. Sometimes it takes the form of books, podcasts, conferences, politics, or consulting. It works because the doubts are not stupid. Change really can produce bad outcomes. It would not be a good business if there were nothing to worry about. But the existence of danger is not the same as the inevitability of it. AI is not actually an independent dangerous being no matter how rational it is to believe it may be. The word AI is not a very useful label. It sits on top of a bunch of more specific technologies. In this debate, we are mostly talking about large language models. They are, in the most literal sense, word counting machines. They train on human content, split language into tokens, and learn statistical relationships between them. When you ask a question, a large statistical engine starts predicting the next word, and then the next, and then the next. That sounds reductive. It is also important. These systems are remarkably good. That is why we use them. But the idea that the model itself has awareness, consciousness, a plan, or a private desire to do anything is mythological. AI is two things at once. It is as dumb as dumb can be, and astonishingly useful at the same time. Both are true. So when I say AI is benign, I do not mean it cannot be used to do harm. I mean it does not originate purpose. Its human users do that. The dangers of AI are human, not AI itself. Jonathan’s questions moved through the most relevant buckets. He is a very good interrogator. Employment. Political disruption. Mental health and cognition. Malicious actors. And then the big one: AI gets smarter than us, develops its own agenda, becomes agentic, and kills us. My answer is not that these concerns are imaginary. It is that the agent in those sentences is not an AI agent, but a human agent. If AI is used in war, the problem is not that AI chose war. Military and political leaders choose to buy, deploy, and authorize systems. If AI floods politics with synthetic persuasion, the problem is not that AI hates democracy. Political actors decided to use cheap persuasive techniques and undermine trust. If AI makes fraud easier, the problem is not that AI became a criminal. Criminals acquired better tools. If AI disrupts jobs, the problem is not that AI dislikes workers. Employers, investors, customers, and governments decide how much labor time to purchase versus machine time. The fact that the dangers are human does not make them smaller. It may make them larger, because humans have an exceptional record of using powerful tools badly before learning how to leverage them. This week’s stories make the point. Alex Chalmers is right that underneath every AI argument there are philosophical disagreements: consciousness, alignment, explanatory knowledge, governance, and whether AI replaces or complements labor. Rohit Krishnan’s artificial-life essay adds a useful twist. Foundation models are not alive, but they are rich substrates. If humans wrap them in goals, tools, memory, selection pressure, and permissions, they can produce behavior that looks inherently adaptive. That is where people get frightened. It is also where we have to be precise. An agent is not magic even though when I use them I often feel as if it is. It is a role, a purpose, a set of rules, a set of permissions, and access to tools. In OpenClaw, that can be as simple as text files: SOUL.md says who the agent is, USER.md says who the human is, and other files define what the agent can remember and do. You can shape an agent to care about anything via these simple text files. You can make it desire money, give it access to an account, and tell it to be ruthless. That would be dangerous. But the danger came from the human who wrote the role, granted the access, and defined the objective. Addy Osmani’s agent harness piece is useful because it makes this operational. The model is only one part of the system. The harness is the prompts, tools, context, hooks, sandboxes, subagents, logs, recovery paths, and observability around it. A decent model with a great harness can beat a great model with a bad harness. That is also where human governance lives - in the agent’s defined nature, goals, tools and so on. In the early days of AI (quite recently actually) I debated Gary Marcus and at that time the debate was whether AI was intelligent. Hallucinations dominated the evidence. The important questions are no longer only whether a model is capable and powerful. They are: who defines it, “runs” it? Who owns the prompt? Who owns the tool permission? Who owns the data boundary? Who is liable when an agent follows instructions correctly and the instructions were reckless? Human accountability for their agents is core. Ben Thompson’s pieces on inference and deployment describe where the value is concentrating. It is not only in training frontier models. It is in systems that run continuously inside real workflows. Anthropic’s OpenClaw reversal shows the same thing from the vendor side. Once agents consume compute, context, tool calls, and long-running orchestration, the platform has to put a boundary around usage and cost. The a16z essays make the enterprise implication explicit: the valuable layer may move from systems of record to systems of intelligence. Read it, it’s good. Different language, same direction. Control layers are hardening. The military case is the cleanest test. AI will be able to target. That is not a question, it is a fact. If a Department of War buys it, of course it will try to use it for war. They are called the Department of War. The question underneath is not what the AI wants. The question is what humans authorize. For the first time, we have a technology that can act faster than we can and do things under our control that carry out both our best and worst intentions. That is an enormous governance problem. It is still a human one. Who controls the humans who use AI? The same pattern appears in softer domains. Brian Merchant’s piece on artists shows that generative AI does not have to hate artists to damage their economics. A client can use it as leverage. A platform can use it to flood supply. A buyer can decide that faster and cheaper beats craft. The machine is amoral. The market behavior around it can still be brutal. Joshua Dzieza’s piece on AI-generated research points to another bottleneck. The problem is not that software wants to corrupt science. It is that production gets cheaper faster than review, replication, and judgment can scale. When fluent output overwhelms verification, the human institutions around knowledge become the weak point. There is a huge discussion in coding around the bottleneck now being human review, and the need for agentic code review. That is right. It will happen. But the goal of a project sets the context for that review, and humans set the goals. Campbell’s bubble essay and the Cerebras IPO show the market version. AI infrastructure demand can be real and still produce overextended prices. Public investors are now trying to buy the compute, chips, power, data centers, and suppliers underneath every agentic workflow. That is rational behavior for a growth-seeking investor. The venture lesson is similar. Cambridge Associates warns that seed is crowded, follow-on conversion is difficult, and dispersion remains extreme. [SignalRank Agent](https://agent.signalrank.com/) checked that warning against the data. That is what good AI should do. It does not replace judgment. It can be programmed to discipline judgment, extend it, and force it to show its work. I created the SignalRank agent. I asked it to try to produce deterministic answers using probabilistic technology. That is the right relationship. My goals, its attempt to meet them. So no, I do not think AI will kill us all by waking up one morning and deciding to become our enemy. Humans of course could kill us. And they might use AI. Could humans use AI in ways that make society more unequal, more passive, more surveilled, more fraudulent, more militarized, more cynical, and less capable of governing itself? Yes. Obviously. Could the use of AI lead to fewer jobs? Yes. The reason I like to distinguish between jobs and work is that fewer jobs is, in my view, inevitable as automation embraces most repetitive work. And a good thing if it means a lower working day and more leisure time. But as jobs disappear, because paid labor can be automated, that is disruptive, and for many people it will be painful. But work does not disappear as paid jobs do. Work is another word for effort. We work on our gardens, even though nobody pays us. We work on our hobbies. We work on travel and entertainment. Even reading a book is work. Humans are constantly reinterpreting the future they want and working to make it happen. The right question is not whether there will be tasks humans undertake. There will be. And we will enjoy the new time we have to undertake that work. The real question is how the wealth created by automation and the decline of paid labor is used to benefit society overall, or as we said last week, civilization. And whether we can build institutions that preserve agency rather than turning abundance into passivity. The open question is not “Will AI kill us all?” as a science-fiction slogan. It is whether humans can govern powerful systems wisely enough to keep them in service of human purpose. If we get that right, AI can amplify human existence. If we get it wrong, AI will amplify human anomie. In either case, the deciding agent is still us.