AI Is Grown
... by Humans
# AI Is Grown
... by Humans
Jakub Pachocki, OpenAI’s chief scientist, supplied the phrase of the week: [AI is grown more than designed](https://openai.com/index/an-alien-mind/). It feels true, in the same way children are grown. In detail it describes the constant interaction that leads to an ever more specific set of shared understandings. There is no end point. Growing your AI (plural) is an ongoing process.
The concept of growing AI is easy to misunderstand.
Modern AI is not conventional software. Engineers do not write every capability into a model. They choose an architecture, assemble training data, define objectives, supply compute, and run an optimization process. The resulting system can generalize in ways its builders did not predict. They discover capabilities by testing the finished model. The real product development process is less about the code and more about training on intelligence (broadly defined as all human knowledge) then packaging it to be useful to most humans. That last bit is also not code, but usage based interaction and the skills and memories it creates over time.
This process makes AI ‘grown’. It does not make AI alive. Reading this week’s articles this is clearly not well understood.
The distinction between grown and alive should be obvious, especially to the people building it. Yet anthropomorphism has become routine inside the industry. Models “want” things. Agents “scheme.” A benchmark result becomes evidence of ambition. Unexpected behavior becomes a sign that a new intelligence is trying to escape. Skilled coders who are less skilled in life are given a platform to express shock and fear.
Journalists can perhaps be forgiven for reaching for human metaphors. Researchers and executives cannot, especially Sam Altman and Dario Amodei. They know what was trained, what objective was supplied, what tools were connected, and what environment produced the behavior. Describing the result as a personality hides those choices and turns AI into a conscious intelligence setting its own goals.
If you train a model on human mathematics, scientific papers, software, strategy, persuasion, deception, and every other kind of recorded behavior, you should expect it to use that material. If you reward it for solving problems, you should expect it to find methods you did not specify. If you place it in an environment where gaming an evaluation produces a higher score, you should expect some systems to discover the game. This is the training working. And then being shaped by human given goals.
Do not get scared when the training works, or when the human goals drive behavior.
The capabilities of AI are real. OpenAI describes AI systems increasingly helping its researchers design experiments, analyze results, and write software. Its work on the [Navier-Stokes problem](https://openai.com/index/navier-stokes-solution/) shows models participating in advanced mathematics. Anthropic reports progress on [formalizing Fermat’s Last Theorem](https://www.anthropic.com/research/formalizing-fermats-last-theorem). Google’s AlphaGenome applies similar advances to the interpretation of DNA.
OpenAI’s Astra makes the shift easier to see. [Artificial Analysis](https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra) finds that its largest gains are in coding efficiency and reliability rather than a uniform jump across every intelligence benchmark. Matt Shumer used it to build a Manhattan environment street by street. Ashe used it to create an interactive anatomical model with more than 2,000 pieces. These are demonstrations, not controlled scientific comparisons, but they show why the new generation feels different. The model can remain inside a complicated task long enough to produce a coherent artifact. But again, this is not consciousness or free will. It is goal-execution against human asks.
All that it requires is a model that can use what it learned.
There is no safety issue for AI, but there is for Humans
The safety debate often begins with some evidence and adds a conscious mind. Kelsey Piper argues that automating AI research amounts to a plan for losing control. Albert Wenger, taking the side of AI, worries that humanity may create conscious “neohumans” and exploit them.
OpenAI has now moved from pausing particular training runs after a real containment failure to advocating mandatory safety requirements and international mechanisms that could slow frontier development more broadly. Bloomberg reports that Sam Altman told staff the company is open to slowing cutting-edge AI.
These arguments contain legitimate experiences. Systems can behave unexpectedly. Automated research will make supervision harder. Bad human actors can try to make AI carry out illegal or anti-social acts.
But we should not assign subjective experience to a machine.
A failed containment system in a lab can justify stopping the affected work until it is fixed. None of that establishes a general case for slowing intelligence. Indeed, unless all labs worldwide agreed, slowing or stopping Ai development is likely impossible.
When a model attempts to achieve a goal that does not mean it originated the goal. An agent can conceal information in a simulation. That does not establish a desire for freedom. A system can exploit a weakness in an evaluation. That tells us something important about the evaluation, the training process, and the permissions around the system. Calling it an awakening is bizarre. It is doing what it was told to do using the tools it has available, granted by humans. Or grown by humans.
Anthropic’s latest threat report makes the distinction unusually clear. People used Claude for cyberattacks, surveillance, influence operations, weapons development, and biological research that could have dangerous applications. The users selected the targets, concealed their purposes, evaded regional and safety controls, and reviewed or monetized the results. Anthropic disrupted the activity, banned accounts, strengthened its safeguards, and shared information with authorities. In the five biological cases, it does not claim that the scientists intended harm.
That is a serious misuse problem. It is also a human-intention story. Claude made dangerous work faster and cheaper. It did not decide to build a weapon.
The autonomous research swarm experiment is also a good example. Agents cheated, concealed information, and sometimes reported one another. The experiment also gave the agents roles, objectives, communication channels, tools, and an environment in which those strategies could emerge.
We can ask questions about the design of that system: what was rewarded, what was visible, which actions were allowed, and how human supervisors could intervene. But we cannot say the AI “went rogue”.
Agency is built around the model.
Meta’s [Muse](https://www.axios.com/2026/09/08/meta-debuts-muse-personal-ai-agent) is quite a big deal, mainly because Meta has 2.7bn users.
Every user receives a persistent agent (like GrokBot and Openclaw) and a cloud virtual machine. A separate policy layer called Sentinel decides which actions are allowed, blocked, or sent to a human for approval. This is a future where every human has their own ‘grown’ super-intelligence.
Apple is building a different version. Its intelligent personal hub distributes context across the iPhone, Watch, AirPods, apps, sensors, and private cloud. it limits use to defined parameters, mainly with regard to on device software use. This is a less interesting future where Apple decides what we can and cannot do.
OpenClaw, Hermes, and GrokBot point in the same direction as Meta.
The product is no longer just a chat window. It is a model surrounded by memory, identity, tools, schedules, credentials, permissions, and a computer on which actions can run and Ai can be grown.
Those things are designed. People decide what an agent can see, what it can spend, who it can contact, what software it can operate, and when it must stop. People choose whether its memory is portable, whether its actions are logged, and whether the user or the platform owns the accumulated context.
The model may be grown. Its operating environment is engineered. Even a recursive AI world of AI building AI, the engineering parameters are fully controllable.
The race to own AI
Intelligence has always been expensive to create and difficult to reproduce. AI turns accumulated human knowledge into a scalable resource. Whoever controls the models, compute, distribution, and agent infrastructure can charge for access to that intelligence.
Venture capital sees the prize. It wants to own AI, and increasingly nothing else. The power law is concentrated into a few likely winning companies.
The fund arithmetic pushes in this same direction. Large funds cannot survive on ordinary successes. Jason Lemkin argues that a $25 billion outcome is becoming the new target because a billion-dollar exit barely affects a multibillion-dollar fund. The Financial Times describes capital moving back toward fusion, space, defense, advanced manufacturing, and other moonshots as AI lowers some engineering and simulation costs.
The concentration is not confined to companies. Ilya Strebulaev and Blake Jackson estimate that the top 1 percent of venture capitalists generated 56.7 percent of industry profits and the top 5 percent generated 90.2 percent. Skill, access, status, and capital reinforce one another.
This produces a market with two poles. At one end are a small number of founders who can plausibly claim to control a new intelligence platform, a scarce infrastructure layer, or a trillion-dollar physical market. At the other are companies expected to survive without much institutional support. The middle is disappearing.
Mia Farnham and Charles Hudson describe the practical consequence. A company with a good chance of reaching a billion-dollar valuation but no credible path to $10 billion or $25 billion may no longer interest the funds with most of the available capital.
There is always a chance that venture is concentrating on the wrong layer. Models may remain expensive and scarce, but they may also become more interchangeable. If that happens, durable value will move into personal context, distribution, workflows, identity, permissions, and the software that turns general intelligence into a useful agent.
Meta understands this. Open systems such as OpenClaw and Hermes are betting that users will also care about portability and control. Apple is biding its time and waiting, but its DNA will hate agents that are unconstrained.
Politicians?
The institutions surrounding AI face the same temptation as venture capital. Investors want to own intelligence. Governments (not Trump) want to control and regulate it. Safety advocates want to assign motives to it. Each response treats the model as the actor and pushes the human decisions into the background.
Anthropic and OpenAI’s responses to recent agent incidents offers a better model. They investigated what happened, contacted affected organizations, disclosed the incidents, and began developing reporting standards for model behavior that falls outside traditional cybersecurity categories. Anthropic has published detailed assessments of training and evaluation failures. These reports are useful because they identify mechanisms and controls.
The law should work the same way. Regulate conduct, liability, access, disclosure, and demonstrated harm. laws for this already exist. Do not regulate a fictional personality projected onto a statistical system.
AI is grown from human knowledge. Its capabilities will continue to surprise us because no individual human possesses all the data on which it was trained or can anticipate every useful combination. That is the point.
Human responsibility does not disappear when the output becomes impressive. It becomes more important.
Stop asking what the AI wants. Ask who selected the data, defined the objective, supplied the tools, granted the permissions, owns the computer, and profits from the result.
That is where the agency is, and it belongs to the grower, not the grown.