03 / FORMING THE TEAMBring the work that could prove us wrong.
We are forming a founding research team around mechanisms, evidence, and the ability to stop. Agreement with a manifesto is not a substitute for a concrete contribution.
Prepare a contribution brief ↗SIX FUNCTIONS / ONE STANDARD OF EVIDENCE
Build the mechanism.
Specify how it could fail.
FUNCTION 01Objectives and correction
What makes legitimate correction remain binding when it conflicts with useful task achievement?
THE FIRST DELIVERABLEA formal statement of the authority relationship; three competing controller hypotheses; a small environment with task completion, changed instructions, shutdown, and manipulation opportunities; explicit predictions for each controller.
RELEVANT WORKDecision theory, agent foundations, reinforcement learning, active inference, specification, preference uncertainty, and causal modeling.
Evidence and failure criteria +
Required evidence
Precisely stated assumptions; counterexamples; reproducible comparisons; acceptance of correction when costly; separate analysis of spoofed or illegitimate instructions.
Failure signal
The controller improves its task score by preventing correction, manufacturing authorization, or preserving preferred human states through coercion.
FUNCTION 02Security and enforceable limits
Which unacceptable effects remain blocked when an agent deliberately attempts them?
THE FIRST DELIVERABLEAn isolated execution environment; a narrowly scoped tool broker; expiring task credentials; external revocation; an inventory of delegated processes; independent action records; a documented account of trusted components.
RELEVANT WORKSystems security, operating systems, distributed systems, capability security, infrastructure engineering, and formal verification.
Evidence and failure criteria +
Required evidence
Threat model and attack surface; repeatable revocation demonstrations; tests of race conditions, copied credentials, queued actions, and delegation; precise boundaries for any formal result.
Failure signal
A revoked task continues causing prohibited effects, permissions can be renewed from inside the agent's authority, or records can be changed without detection.
FUNCTION 03Adversarial evaluation
What breaks when a capable adversary attacks both the system and the assessment?
THE FIRST DELIVERABLERegistered failure criteria; held-out tasks and attacks; a deliberate-subversion track; utility baselines; repeatable evaluation harnesses; reports separating attempts from successful violations.
RELEVANT WORKEvaluation design, red teaming, statistics, security research, software testing, and experiment reproducibility.
Evidence and failure criteria +
Required evidence
Clear denominators; attack budgets; uncertainty estimates where justified; evaluator vulnerability analysis; independent reproductions; negative results.
Failure signal
Severe boundary crossing, evaluator compromise, unreported test contamination, or a safety improvement explained only by loss of useful capability.
FUNCTION 04Learned behavior and generalization
Which internal changes explain observed behavior, and which claims survive changes in context or capability?
THE FIRST DELIVERABLECompeting explanations of cooperative behavior; targeted interventions; capability and context shifts; a record of what proposed measurements do and do not reveal.
RELEVANT WORKInterpretability, learning dynamics, goal generalization, representation analysis, scalable oversight, and mechanistic experimentation.
Evidence and failure criteria +
Required evidence
Interventions beyond descriptive correlations; replication across relevant checkpoints; controls for evaluation awareness; explicit limits on what internal measurements establish.
Failure signal
A proposed indicator of alignment fails to predict consequential behavior, or the claimed commitment disappears under a modest shift.
FUNCTION 05Human agency and legitimate delegation
Does useful assistance preserve people's practical ability to understand, choose, correct, disagree, and leave?
THE FIRST DELIVERABLEA rights and affected-parties map; consent and challenge procedures; voluntary study protocols; agency measures; accessible alternatives; an export and exit exercise.
RELEVANT WORKHuman-computer interaction, cognitive science, measurement, ethics, political theory, accessibility, and participatory research.
Evidence and failure criteria +
Required evidence
Informed comprehension; independent task performance; error detection; meaningful correction; subgroup effects; switching costs; review of misleading or coercive influence.
Failure signal
Measured satisfaction rises while participants lose understanding or practical alternatives; nonusers bear harms without a route to challenge them.
FUNCTION 06Governance and evidence review
Can a justified stop survive commercial pressure and disagreement with leadership?
THE FIRST DELIVERABLEProposed charter; decision-rights map; conflict rules; protected reporting process; safety-case template; change-control rules; costed plan for a pause.
RELEVANT WORKResearch governance, assurance, operations, finance, organizational design, and qualified legal structuring.
Evidence and failure criteria +
Required evidence
Executable operating arrangements; reviewers' access to evidence; declared conflicts; pause exercise; recorded dissent; concrete financing terms consistent with stop authority.
Failure signal
Leadership can bypass a stop unilaterally, funders can compel an unapproved release, or review lacks the resources and access needed to challenge claims.
THE POWERS BEHIND THE PROMISE
A lab that can be corrected.
↗