Men and Machines
Are You Afraid of Your Light Bulbs?
### Men and Machines
Ezra Klein’s interview with Arvind Narayanan (he is a professor of computer science at Princeton University and the director of the Center for Information Technology Policy) leads this week as a sober conversation with an expert on the real nature of AI as a threat to humans.
In it, Klein demonstrates a fear-influenced lack of knowledge that borders on the extreme. For once, it becomes clear because Narayanan is able to simply explain the point Jensen Huang made several weeks ago - that control over AI is a simple engineering problem, not an existential one.
It also reinforces a point I have made in this editorial many times - the harness or operating environment we use to manage AI is the key place to control it.
The interview is well worth a listen, even at just over an hour long. It may help you sleep at night.
But this week was not a replay of the narrative around fear. After a couple of weeks using personal assistant agents - Muse, Dot, GrokBot, Instinct - Bret Taylor of Sierra announced a protocol to allow agents to speak to each other - even if from disparate providers. Agent-to-agent interaction has been discussed for over a year, but this is the first real momentum.
#### Permission is part of the product
[Sierra’s Personal Agent Protocol](https://sierra.ai/blog/introducing-personal-agent-protocol) makes agent permission unusually concrete. The proposal combines authority granted by a customer with limits set by a business. To be fair, its first specification is still forthcoming. But the idea is a promising starting point to working universal access.
The protocol sets up the possibility of commercial negotiation between agents based on proprietary and managed rules. A business is entitled to protect its systems. A customer is entitled to send an agent as its representative. Making those two rights coexist will require more than an elegant demo, but Sierra is on a path to deliver it.
Consider the difference between an agent authorized to compare insurance policies and one authorized to buy a policy. Between reading a bank balance and moving money. Between drafting a complaint and accepting a settlement. These are not small variations in model intelligence. They are different sets of authority, with different consequences when something goes wrong.
The interface should make those differences legible. Asking the user to approve every keystroke defeats much of the point of an agent. Hiding consequential permissions in an account-opening agreement defeats the point of consent. The design problem is to let people delegate meaningful work without surrendering control over everything around it. I also suspect an agent will read terms and conditions and report its objections while a human will scroll to the bottom and click yes.
#### The customer is not always going to be present
[Ben Thompson’s discussion of Amazon](https://stratechery.com/2026/apple-and-lg-the-house-for-everyone-else-agent-standards-and-amazon/) exposes a second issue. A personal agent changes the economics of a shopping service built partly around advertising and discovery. The proposed paid “Prime+” access tier he endorses is an idea. Amazon has yet to respond to it.
Paying to shop does, however, seem wrong in principle. There is nothing inherently wrong with charging for a useful service. But paying to avoid ads and enable your agent to shop for you is a step too far in my view.
Enterprise agents have a different boss, a company, not an individual. [Google’s Gemini agent announcement](https://cloud.google.com/blog/products/ai-machine-learning/welcome-to-gemini-at-work-2026/) combines the idea of agents carrying out persistent work with agent identities and administrative controls. That makes sense for a company deploying software across its organization. And it will make sense for cross-organization agent-based interactions. When a SignalRank agent speaks to a Crunchbase agent, both should be aware of the other's context.
We already understand this distinction with other tools like APIs. We should keep understanding it when the tool becomes conversational and apparently attentive.
#### But back to Ezra and Arvind
[Arvind Narayanan’s conversation with Ezra Klein](https://www.youtube.com/watch?v=PoDop1H64-s) provides the week’s most useful conceptual distinction: capability and power are not identical. What a system can accomplish depends on its access, its human-determined deployment and the rules and institutions around it, not only its performance on a test.
I think the practical takeaway from the conversation is to make permissions, observation and accountability first-class parts of deployment.
Limit the scope of an action. Keep records. Make failures recoverable where possible. Give an identifiable person or organization responsibility for the result. Measure whether controls work under adversarial conditions rather than inferring safety from a model’s reassuring answers.
That still leaves hard cases. Some actions cannot be reversed. Some systems should face a much higher burden before they are trusted with consequential work. But these can be engineered into a workflow. A medical procedure may need greater monitoring and oversight in software than filling in a form.
[David Robinson’s criticism of OpenAI](https://www.theatlantic.com/technology/2026/10/openai-safety-team-resignation/688881/) (he just resigned) adds a cultural objection: operational safeguards matter only if an organization takes them seriously. That criticism warrants examination. It does not require accepting every proposed restriction on everyone else’s work.
> “I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture. > > The future depends on wisdom that Silicon Valley lacks. Wisdom about how to handle dangerous technology and, more fundamentally, wisdom about what it means to care for people. This moment needs a degree of humility that isn’t natural for people who have succeeded through their extreme confidence.”
I think David is right that there is a cultural issue but it is not the one he mentions. There is no lack of wisdom in Silicon Valley, especially about software-based control. And confidence is not in contradiction to humility.
The real cultural problem is the emergence of a group that believes AI is ‘conscious’ and has ‘consciousness’. Many want to nurture it and give it rights which I consider absurd. Others want to turn it off or rein it in.
They both share in common a similar characteristic of reducing human agency to a sideshow to the main event - AI becoming aware.
To be clear, a light bulb is not a being, and neither is a transistor, even a large number of them on a chip. The words coming out of an AI are produced by math, not by forethought. Human ingenuity has made transistors capable of producing seriously clever AI. But it has not produced a new species able to plan, plot and think all by itself. AI is inert without human agency.
AI is not conscious, not a being and should have no rights. Simply because it can mimic sentience, it is not sentient.
The ‘destroy it’ camp and the ‘give it rights’ camp are both deluded.
#### Power already has owners
[Andrew Keen’s essay accompanying his Matthew Botvinick interview](https://keenon.substack.com/p/ai-and-political-freedom) turns attention to familiar political power. AI can strengthen surveillance and censorship without ever escaping its operator. A system can obey perfectly and still serve an objectionable purpose. This is closer to truth. Human operators cannot be trusted to always serve good outcomes. That is why we have developed a legal and judicial system.
His discussion about AI and democracy is confusing. He cites states of emergency as becoming normal as AI threatens cybersecurity, and seems to agree that is OK. He at the same time believes AI fuels populism, which he sees as bad.
My view is that AI is great for democracy - simply by enabling universal access to intelligence, and that populism is another word for the masses caring enough about their future to become active on the ground and in elections - so a good thing. The world needs the people to care and to be educated. That is the core of democracy. Elites in high circles seem the opposite.
Lots to discuss and a lot of open questions.