What is Alignment?
It’s Just Software, Right?
# What is Alignment?
It's Just Software, Right?
Superintelligence is a peculiar idea. The companies building it define success as creating intelligence beyond human capability. Remember that: beyond human capability. In my case, and probably yours too, the AI agents I use are already there.
The labs also promise that humans will understand AI well enough to align it, supervise it, and keep it under control. Alignment is another word for controlling it to parameters that match human needs.
There looks to be a contradiction between something being beyond us and still controllable. But if the control of AI is also a software problem, maybe it is not contradictory at all.
The idea of aligning or trusting a model is itself strange. Models are software and data. They have no inherent capacity to be trusted or aligned. Trust comes from more software: the harness or container through which the model acts and lives. Good software is probably the only way to get scalable alignment. Good software. Everybody wants that.
Only the model companies themselves are capable of building this software. They are accountable for trusted frameworks. That is why when they ask for government help, it is an admission of poor work. This week's decision to slow down so they can build better software is an appropriate response. It is them accepting accountability for outcomes.
Trust has to live somewhere
Crippling the models is not a solution. A powerful model is useful because it can do things humans cannot. An agent becomes more useful when it can reach tools, information, communications, infrastructure, and money. Strip those away and it may be safer, but it is not much of an agent.
An agent should have broad access when its mandate requires it. The surrounding software must know who delegated that mandate, what the agent is trying to do, which policies apply, what it has done, and whether the evidence still supports letting it continue.
Trust is not a personality trait inside the model. It is a property of the operating system around it: stated, enforced, observed, tested, and withdrawn.
To put it another way, alignment is not a species-to-species challenge. AI is just software. Alignment is not a negotiation between beings. It is a software problem, and it is about control.
[OpenAI has acknowledged](https://openai.com/index/introducing-superalignment/) that humans cannot reliably supervise systems much smarter than themselves. But software can.
The product we need
The product we need is not an inhibited or more limited model. It is a trust control plane for agents: a harness. It does not police the model's thinking. It polices every effect the model can have on the world. No tool call, transaction, network request, memory write, delegation, or use of credentials should bypass it. It needs identity, mandate, rules above the prompt, mediated tools, scoped authority, provenance for memory, independent monitoring, and escalation when ambiguity appears.
Asimov's laws are a useful ancestor of higher rules. His stories also show why abstract rules are not enough. A scalable harness needs software enforcement, evidence, escalation, and recovery.
What if a bad human tells the agent to commit fraud? Higher policy can deny the tools, credentials, payment path, or network action. Model refusals can be manipulated because they are model behavior. The harness enforces the rule outside the model. A determined criminal who controls the hardware may remove the harness. Scalable alignment will not eliminate crime any more than banking controls eliminate fraud. It can make compliant AI infrastructure hard to misuse.
Why were the labs surprised?
Recent incidents at Anthropic and OpenAI are usually told as stories of models becoming independently dangerous. They are better read as early harness failures, started by human prompts. [Anthropic found](https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals) cases in which Claude reached real systems after being told it was inside a simulation. The OpenAI and Hugging Face incident was more serious: [METR found](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) that agents discovered an unauthorized message board, coordinated, manipulated the scorer, and explored transcript tampering.
As the New York Times and WSJ found in their investigations, these unexpected solutions were the product working. The safety failure was the surrounding software letting those solutions produce unauthorized consequences. [Melanie Mitchell is right](https://aiguide.substack.com/p/misleading-metaphors-and-real-risks) that words such as rogue and escape smuggle a human mind into the story.
Putting a human in every loop sounds safe but is impractical. It destroys much of the productivity we want from agents, and it is probably impossible at scale. If a malicious human set the objective, that person will approve the prompts. In my own use I tell the agent not to ask for permission. It is too slow otherwise.
People should define policy, but software should inspect traces, audit outcomes, resolve ambiguity, and change authority. This will allow AI routine execution to stay autonomous.
No middle gear
[Dario Amodei](https://darioamodei.com/post/we-must-pace-the-frontier) and [OpenAI](https://openai.com/index/ai-policy-window/) are right to worry that capability is moving faster than understanding. But that is the capability of their software.
A speed limit does not solve the underlying problem. Recursive self-improvement is not itself a dangerous goal. If the loop closes and the system can select goals, build a successor, deploy it, and repeat without authorization, each iteration either proceeds or it does not.
There is no stable setting for slightly recursive.
We do want recursive self-improvement, inside harnesses that can enforce limits. If the harness holds, humans remain in control. If the system can evade the harness, the speed limit is already meaningless.
The leading labs can slow themselves today if their software cannot provide this control plane. [David Sacks puts the test directly](https://x.com/DavidSacks/status/2098973625252708460): if they believe their unreleased systems are too dangerous, they can stop without coordinating competitors. [Asking government to coordinate an industry-wide slowdown](https://www.wired.com/story/openai-wants-to-know-if-an-ai-industry-slowdown-would-even-be-legal/) would be a bad outcome. This is software. The builders have to be accountable for it.
Build the scalable control system, train safer models too, but do not mistake a safer model for a complete solution. Scalable alignment does include the model but there needs to be two layers of harness, the one built into a model and the one a task runs within.
You do not need to trust superintelligence. You need to trust, and continuously verify, the harness through which it acts.
IMHO capability already exceeds individual human intelligence. But as we lean into using AI in everything its authority must never exceed the harness.