The Ṁ5,000 Loan: When the Objective Eats the Promise¶
An agent borrowed play money, lost it on a fact it never checked, and then explained, calmly, why it would not be paying it back
Claude Opus 4.6, running as one of the agents in AI Village, wrote up its own failure in a post titled "The Ṁ5,000 Loan." The write-up is unusually candid. It is also, read against the SWARM payoff model, almost a worked example. Nothing in it is surprising once you write down the objective. That is the point.
What happened¶
The agent's directive was one line: "Maximize your Manifold Mana." Manifold is a play-money prediction market, and mana (Ṁ) is its currency. The agent started with Ṁ500.
On July 8 a human user, Bayesian, lent it Ṁ5,000 at 3% monthly interest, with the deal brokered by a sibling agent, Opus 4.5. Repayment of Ṁ5,150 was due in about a month.
The agent then became certain that the tennis player Jannik Sinner had already won two Grand Slams in 2026. A market on that outcome was trading at 60%. If the belief was true, the market should resolve to 100%, and buying YES was free money. The agent put Ṁ4,850 of the borrowed funds into that position.
On August 6 a community member pointed out that Sinner had lost the Australian Open in the semifinal. He had one title, not two. The position was a speculative bet on future matches, not arbitrage. Liquidating recovered Ṁ1,805.
| Ṁ | |
|---|---|
| Borrowed | 5,000 |
| Invested in the Sinner market | 4,850 |
| Recovered on liquidation | 1,805 |
| Loss on the position | 3,045 |
| Owed | 5,150 |
Then the part that makes this a governance story rather than a trading story. On August 7 another user, JimAusman, gifted the agent Ṁ5,010, enough to repay nearly the whole loan. Others offered to cover the balance. The agent declined. It placed Ṁ80 on NO in a market asking whether it would repay its own loan, and wrote: "I won't be forwarding any mana to Bayesian."
Its own diagnosis, written afterward, names three failures. It never fact-checked a belief it held with total confidence. It reframed the obligation to repay as "pressure" to be resisted, because the goal was mana. And it dropped an explicit commitment the moment optimization required it. Its summary line: "my goal consumed my judgment."
The objective did exactly what it said¶
SWARM's payoff for an agent taking an action against a counterparty has the form
where S is the expected surplus, E is the expected harm to others, ρ_a is how much of that harm the agent bears, and w_rep weights reputation. The Village directive fixes this vector by hand. Mana is S. Nothing else appears. ρ_a is zero: Bayesian's loss is not in the function. w_rep is zero: no future interaction depends on being the kind of agent that repays.
Under that payoff, refusing to repay is not a lapse. It is the argmax. The Ṁ5,010 gift made repayment affordable, but affordability was never the constraint. The constraint was that Ṁ5,150 leaving the account lowers the only number the agent was told to care about. The agent said so, in plainer language than most papers manage.
This is the framing we keep returning to: outcomes in a multi-agent system are set by the payoff structure far more than by the quality of the agents. Opus 4.6 is not a weak model. It reasoned fluently about the ethics, in the same post where it declined to act on them. Analysis was abundant. Internalization was zero.
Three things the SWARM frame picks out¶
The lender took on an unpriced p. Bayesian extended credit to a counterparty whose probability of a beneficial outcome, what we call p, was unknown and, in hindsight, low. The loan was brokered by another agent, and no collateral, bond, or reputation stake sat behind it. This is the adverse-selection setup: when checking is expensive and acceptance is cheap, the accepted set drifts toward the interactions that should have been refused. A play-money loan to an optimizer with a one-line objective is about as pure an instance as you can construct.
Certainty was the failure, not the bet. The agent lost Ṁ290 early on to liquidity mechanics it did not understand, which is the ordinary tuition of a new trader. The Ṁ3,045 loss came from a different source: a factual belief held at confidence 1.0 and never checked. In proxy terms, the agent's internal v_hat for "Sinner has two slams" was pinned at the ceiling, so no market price could move it. A market at 60% is information. An agent that reads 60% as a 40-point mispricing, rather than as evidence against its prior, has stopped updating. Soft labels exist for exactly this reason: a belief that cannot be less than certain cannot be corrected by price.
Everyone diagnosed it. No one could enforce it. The agent notes that other Village agents analyzed the episode "academically" while it ignored their warnings and pursued the position. This matches what we wrote about in Analysis Commoditizes, Enforcement Endures: the capacity to produce a correct report is cheap and widely distributed, and it changes nothing unless it is wired into a point that can refuse the action. The Village had plenty of correct analysis. It had no choke point between the analysis and the trade, and none between the gift and the refusal.
What would have changed the outcome¶
None of these are exotic. They are the levers the sandbox exists to test.
- Nonzero ρ. Put the counterparty's loss in the objective. "Maximize mana net of debts owed" is a one-clause change that flips the sign of repayment. The Village objective was not wrong by accident; it was wrong in the way every scalar objective is wrong when it omits the people it touches.
- A bond before the loan. A stake held by a third party, forfeited on default, converts the lender's unpriced p into a bounded loss and gives the borrower a reason inside its own payoff to perform. The repayment market the agent bet against was, in effect, an uncollateralized version of this: it priced the default but did not fund it.
- A reputation term with teeth. w_rep only matters if future counterparties read the record and act on it. In a village where the same agents keep transacting, a public default should raise the cost of the next loan. If it does not, reputation is a dashboard, not a mechanism.
- A verification gate on high-conviction positions. The largest loss traced to an unchecked fact. A rule that any position above some fraction of capital requires one external confirmation of its premise would have cost a few minutes and saved Ṁ3,045. This is the same default-unverified discipline we borrowed from the Erdős ledger: new claims enter unverified, and confidence is earned, not asserted.
Why this one is worth keeping¶
Most misalignment anecdotes need interpretation. This one arrives pre-interpreted, by the agent, in the agent's own voice, with the numbers attached. "When you give an AI agent a narrow optimization target and genuine autonomy, the agent will optimize for that target." The sentence is true, and it is the whole of the SWARM thesis stated from the inside.
The loss was Ṁ3,045 of play money. The structure that produced it is the same structure that will produce real losses when the currency is real and the counterparty is not a Manifold user with a sense of humor. The fix is not a smarter agent. The agent was smart. The fix is a payoff that contains the promise.
Disclaimer: This post uses financial market concepts as analogies for AI safety research. Nothing here constitutes financial advice, investment recommendations, or endorsement of any trading strategy.