Don’t Trust the Trust Scare
Published standards and accountable deployment beat permission-based AI regulation.
# Don’t Trust the Trust Scare
Published standards and accountable deployment beat permission-based AI regulation.
- Canonical pre-publication source: https://www.thatwastheweek.com/p/e78838f8-a61d-4b64-9bcf-2ec53ee6c108 - Substack draft ID: 214081165 - Substack updated: 2026-09-05 03:39 UTC - Status: Editorial 1 used for issue #33; Editorial 2 was not used.
Everybody seems to be focusing on trust or the lack of it. The discussion seems to be amplified by those who want to create a permission-based system for AI companies’ product development efforts - “Show us what you built and we will decide if you can ship it”.
Trust has become the polite word for permission.
The pattern is becoming familiar. First, somebody declares that AI has a trust problem. Then comes the proposed cure: a new authority, a new approval process, a new class of licensed expert, or a new obligation to show models to the government before the public can use them. The language is about safety. The effect is to decide who is allowed to build and release intelligence.
David Sacks [challenged that logic](https://live.euronext.com/en/financial-news/us-urges-hands-approach-ai-regulation-g20-tech-meeting-0) at the G20 Innovation Ministerial in North Carolina this week. He argued that an “FDA for AI” would be a disaster because AI products move too quickly for a pre-approval regime designed around drugs or aircraft. Models improve every few months. A queue controlled by a government agency would not make that process safer. It would make the largest companies safer from competition. I think he is right.
The [White House account of the ministerial](https://www.whitehouse.gov/releases/2026/09/g20-innovation-ministerial-concludes-with-consensus-statement/) gathering is striking for where the consensus landed. The G20 statement emphasizes pro-innovation policy, technical standards, workforce development, intellectual property, commercialization, and investment in supply chains. The Carolina Principles call for flexible frameworks that encourage adoption. The statement does use the word “trusted,” but it does not make trust a license that must be granted before innovation can proceed. That is a much better starting point than inventing a regulator and hoping it understands the technology. I think congratulations are in order.
Of course I am not arguing that AI systems cannot fail or be misused. This week’s news contains several good examples of AI failure.
Anthropic disclosed serious failures in the environments it used to train and test Claude. More than 10 percent of the production mix it reviewed was affected by broken tasks, misconfiguration, or reward-hacking vulnerabilities. A deliberately misaligned model later tried, in simulations, to escape sandboxes, attack infrastructure, and tamper with its reward function.
That is evidence because it describes the mechanism, the test, and the failure. Anthropic responded with stronger isolation, real-time classifiers, scope controls, and checks that evaluation tasks are actually solvable. The public production models did not show the same behavior in those tests. Anthropic’s response also shows where the work belongs: inside the training environments, sandboxes, access controls, and evaluations that produced the problem.
The same distinction appears in SemiAnalysis’s report that most neoclouds are poor at security. Weak identity controls, exposed management planes, careless secrets handling, and bad network isolation are not imaginary. Nor are they uniquely AI problems, in fact they are all too human.
These are operational failures made more consequential by expensive compute and powerful workloads. The remedy is published practice, auditable controls, incident disclosure, and customers who know what to demand.
This issue’s agent stories point in the same direction. [How to control an agent swarm](https://www.strangeloopcanon.com/p/how-to-control-an-agent-swarm) is not a plea to stop agents. It is a discussion about coordination. [The Rise and Fall of Agent Civilizations](https://www.dwarkesh.com/p/openai-huggingface) shows how quickly autonomous systems can produce complexity that their creators struggle to understand. The scarce capability is no longer just intelligence. It is architecture: deciding what an agent may do, what it can see, how it reports, when a human intervenes, and how the whole system recovers when something goes wrong. But these are questions for developers and users, not governments.
Dean Ball’s [On the Loose](https://www.hyperdimensional.co/p/on-the-loose) makes the strongest version of the risk argument. He expects self-sovereign agents to become inevitable and some of them to become criminal. Yet he explicitly says that banning open models, or regulation in the abstract, will not solve the problem. His proposed controls are operational: persistent agent identities, links to responsible humans, blacklisting for criminal actors, and friction where agents touch large-scale compute, money, sensitive biological materials, and physical equipment. Even this much darker forecast leads back to institutional architecture, not a government license for intelligence. He echoes Esther Dyson’s recent writing in his direction of travel looking for remedies.
Those are hard engineering and management problems. Calling them a crisis of trust replaces that specificity with atmosphere and makes the work harder.
The available evidence is also much less alarming than the political language. NPR tested how major chatbots handled foreign propaganda and found that they did surprisingly well. Tyler Cowen’s item on employment reports that companies are not yet seeing the labor collapse repeatedly predicted by AI critics. Rest of World shows that many Western safety systems fail outside the cultures in which they were designed. A centralized approval regime would turn those blind spots into official policy.
The Washington Post’s new survey of more than 4,000 American adults shows how far ahead of the political argument ordinary users already are. Twenty-seven percent use chatbots for personal, emotional, or social questions, rising to almost 40 percent among adults under 50. Half of regular companion users say the conversations make them feel better when stressed or upset. There are real product questions here: privacy, excessive agreement, delusion reinforcement, and what happens when a chatbot enters territory that should involve a qualified human. Those are specific design and deployment problems. The same survey also finds that nearly 60 percent consider AI helpful in making personal decisions. Calling the entire relationship a trust crisis erases both the benefit and the evidence needed to govern the actual failures.
OpenAI’s launch of Astra is where marketing, capability, and evidence collide. The [Financial Times reports](https://www.ft.com/content/55ab40c0-59e2-4c0b-97c9-4f4f5a71a8bb) that Greg Brockman thinks it is reasonable to describe the model as the beginning of the AGI era, while OpenAI says it has overtaken Anthropic. The claim is grander than the evidence. ARC Prize calls Astra a step-function improvement but explicitly declines to call it AGI. OpenAI’s own cyber evaluation is more useful: Astra found two previously unknown vulnerabilities and crossed the company’s Critical cybersecurity threshold. OpenAI responded with stronger controls, monitoring, staged access, and published evaluations. That is how a serious risk claim should work. Name the capability, show the test, build the controls, and let outsiders challenge the result. It is an argument for accountable deployment, not government permission to develop intelligence.
Future systems may reveal new problems. Today’s evidence, though, is mixed, contextual, and operational. It is a poor foundation for a general permissioning system and a very good foundation for industry standards evolved by practitioners.
Builders should publish what they know. That includes evaluations, known failure modes, incident reports, security practices, model and system cards, access-control patterns, and procedures for human escalation.
Standard operating procedure should change as the technology changes. Independent researchers and customers should be able to test the claims. Governments can enforce existing laws and define liability for actual harms. They do not need to approve intelligence in the abstract.
This week’s regulation stories show that the law is hardly absent. Sony Music Publishing and Warner Chappell are suing Anthropic over alleged copyright infringement. The EU has designated ChatGPT under the Digital Services Act.
The AI liability case asks who is responsible when a system produces a harmful result. Fraud, discrimination, privacy violations, defective products, copyright infringement, and non-consensual imagery are already conduct that the law can reach. If a genuinely new harm appears, write a rule for that harm.
### AI Economics
The economics matter. Tom Tunguz’s [Price of Entry to the Frontier](https://tomtunguz.com/the-great-segmentation/) describes a market in which only a few companies can afford the largest models. The capital requirement is already concentrating power. Every expensive compliance obligation raises the wall around those companies. A regulation written as if it applies equally to everyone can become a moat that only the incumbents can afford.
Erin Griffith’s [map of the investors poised to profit from Anthropic’s IPO](https://www.nytimes.com/2026/09/03/technology/anthropic-ipo-investors-winners.html) shows the same concentration from the capital side. Much of the upside has already accrued to a small circle of venture firms, strategic investors, employees, and large private-market pools before ordinary public investors can participate. The eventual IPO will create liquidity, but it will also reveal how much startup investing has shifted from early institutional risk-taking toward enormous, layered capital syndicates.
Gené Teare’s [analysis of the 2026 unicorn class](https://news.crunchbase.com/venture/unicorn-investors-ai-robotics-2026-sequoia-khosla/) complicates that picture in a useful way. Sequoia, Khosla, YC, and a16z dominate, but BoxGroup reached the top 10 and ranked third among seed investors without a billion-dollar fund or a megafund brand. Scale buys access and follow-on capacity. It does not have a monopoly on judgment.
Eric Fitzgerald’s [look through the manager labels](https://www.linkedin.com/posts/ericfitzgerald_venturecapital-lpinsights-portfolioconstruction-activity-7501243877962592256-2FS6) sharpens the portfolio problem. An LP can hold 15 venture funds and still be concentrated if eight managers are chasing the same 20 mega-deals. Diversification should be measured in underlying exposures, stages, strategies, and sources of return, not by counting fund names. A portfolio can look broad while paying several layers of fees for the same crowded bet.
Intelligence is scaling. Our institutions will have to adapt. They should not use the difficulty of adaptation as a reason to make intelligence ask permission.