Capability and governance are different properties. A system may plan accurately, use tools reliably, and adapt to changing conditions while still pursuing the wrong objective, acting beyond its authority, or making failures difficult to detect.
I previously described capability as no longer the hard part. That was too broad. Capability remains difficult and uneven. The more precise claim is that greater capability does not resolve the governance problem. It often enlarges it by increasing the range and consequence of actions a system can take.
Autonomy therefore needs more than intelligence. It needs an operating agreement about authority, observation, intervention, and responsibility.
Governance begins with authority
Before asking whether a system can act, ask whose intent authorizes the action. A user may delegate a bounded task. An organization may define a standing policy. A publication may allow a pipeline to prepare drafts but reserve indexing for an editor. These are different grants of authority.
The system should not infer that permission in one context applies everywhere. Permission to generate text is not permission to publish it. Permission to recommend is not permission to transact. Permission to optimize one metric is not permission to trade away every value that was omitted from that metric.
Good governance makes scope explicit: which resources, decisions, audiences, time periods, and consequences fall within the grant. It also specifies who can revoke it.
Observability is more than logging
A long activity log can record every event while explaining very little. Useful observability connects behavior to intent. An operator needs to know what changed, which signal contributed, which rule or objective applied, and whether the result remained inside the permitted boundary.
The level of explanation should match the consequence. A low-risk formatting decision may need only a reproducible record. A public claim, financial action, safety recommendation, or persistent policy change deserves stronger evidence and a clearer path back to accountable people.
This is why provenance matters. The W3C PROV model treats the origin of an object through the entities, activities, and agents involved in producing it. In an autonomous system, provenance helps distinguish an output from the process and authority that made it possible. It does not explain every internal computation, but it gives governance something stable to inspect.
Intervention must be designed before failure
“Human in the loop” is too vague if nobody knows where intervention occurs. A person may approve every action, review only high-risk cases, supervise performance, or remain available for emergency override. Each arrangement produces different latency, workload, and risk.
The appropriate design depends on consequence and reversibility. A system can handle routine, low-impact transformations with minimal attention if failures are easy to detect and undo. Actions with broad or irreversible effects need earlier or stronger intervention.
NIST's AI Risk Management Framework organizes risk work around governing, mapping, measuring, and managing rather than treating review as one final checkpoint. Its generative-AI profile notes that some applications may require additional human review, tracking, documentation, and management oversight. The important word is additional: oversight should respond to the context of use, not exist as a decorative approval label.
Reversibility changes what autonomy can safely attempt
A reversible system can explore within boundaries because errors do not automatically become permanent commitments. Reversibility may mean restoring a previous version, withdrawing a publication, cancelling a pending action, reverting a policy, or isolating the part of a process that failed.
Rollback is not sufficient on its own. If nobody notices the failure, or if the original state cannot be reconstructed, a theoretical reversal mechanism offers little protection. Reversibility depends on versioning, provenance, monitoring, and an operator who has both authority and a usable way to intervene.
This project made that distinction explicit by separating generated, reviewed, and published states. A draft can exist, render, and be tested without entering indexing. Promotion is reversible because it is a separate state transition rather than an accidental consequence of saving. Generation Is Not Publication describes the editorial version of this rule.
The objective is part of the risk surface
Autonomous systems are often discussed as if risk begins when execution goes wrong. A system can execute perfectly against a harmful or incomplete objective.
If a publishing pipeline is rewarded for volume, it can produce more pages while reducing the attention given to each one. If it is rewarded only for clicks, it can learn to overstate. If a recommendation system optimizes engagement without representing user welfare, the feedback loop may amplify material that succeeds by exhausting or provoking people.
Governance therefore has to inspect the target, not only the mechanism. What is being optimized? Which values are constraints rather than metrics? Who bears the cost of a local improvement? What observation would show that the objective itself needs revision?
Evaluation must continue after deployment
A pre-release test samples expected conditions. An autonomous system is valuable precisely because conditions change. Evaluation must continue while the system operates.
That does not mean blindly optimizing from every new signal. Measurements can be noisy, delayed, or shaped by the system's own behavior. The purpose of feedback is first diagnostic: reveal drift, unexpected consequences, coverage gaps, or repeated interventions. Policy changes should preserve the reason and evidence behind them.
The same principle applies to content. Search performance can identify articles that are not being found or indexed, but it cannot decide whether the correct response is more content, better evidence, a clearer structure, or no article at all. An autonomous loop should surface the decision, not hide it behind automatic expansion.
A practical governance contract
For the systems described on this site, I use seven questions:
1. Intent: what durable purpose is the system permitted to pursue?
2. Authority: which actions and resources fall within that permission?
3. Evidence: what observations may justify a decision?
4. Visibility: what must be recorded and understandable to an operator?
5. Intervention: where can a person pause, reject, redirect, or narrow the process?
6. Reversal: how is a previous safe state restored?
7. Learning: how does a failure become a changed constraint rather than a forgotten incident?
These questions do not guarantee trustworthy behavior. They make trust testable. A system that cannot answer them may still be impressive, but its autonomy is operationally immature.
Intelligence expands possibility; governance preserves legitimacy
The purpose of governance is not to make autonomous systems timid. It is to let them act within a mandate that remains legible to the people affected by it.
The 10 Directions of Autonomy states observability, reversibility, and human override as requirements. The experience of building the publishing system has strengthened that position. Intelligence can propose, synthesize, and adapt. Governance decides what counts as authorized progress and keeps a path open when the system is confidently wrong.
Autonomy without governance is not independence. It is unbounded delegation.