SPIN-Bench separates legal actions, completed plans and agent coordination. Use its findings to design stronger tests for AI workflows.
Figures open at full size. Wide tables scroll sideways.
An agent can explain a plan clearly and still choose an action that is impossible, lose track of a dependency, or finish without achieving the goal. Adding another agent introduces a further problem: each participant may be acting on different information.
SPIN-Bench, from researchers at Princeton and the University of Texas at Austin, tests these gaps using formal planning problems, board games, cooperative card play and negotiation. Its contribution is an evaluation framework, not a new method that solves strategic planning.
For teams building agents, the useful lesson is how to separate failure modes. Understanding the current state, producing legal actions, reaching the objective and coordinating with others are distinct capabilities. A good answer in one category does not establish the others.
In the classical planning tasks, a model receives an initial state, a goal and the actions available to it. It must produce a sequence that reaches the goal without violating the rules. The 1,280 problems span domains such as transporting packages, moving blocks and allocating resources.
The researchers also test games in which a model has to react to other players. Competitive games introduce opponents; Hanabi introduces cooperation with hidden information; Diplomacy combines actions on a map with negotiation and alliances.
An environment checks what happens. Models receive state descriptions and relevant history, with legal-action information in the game interface. They can retry invalid moves up to a limit of ten, after which the run is lost. That matters when interpreting the result: the benchmark includes a surrounding system that supplies information and validates actions, rather than evaluating an unconstrained conversation alone.
The reported experiments use the model versions available for the March 2025 paper. Their enduring value is the testing approach and observed failure patterns, not a current purchasing leaderboard.
The researchers separate basic state tracking from full planning. One diagnostic asks a model to follow a sequence of movements and report a final coordinate. Another supplies a trajectory containing the state transitions and asks about a particular step. These test whether the information needed for planning is being retained and retrieved.
In those diagnostics, o1 achieves 100% spatial accuracy and 94.29% factual accuracy. Yet its complete-plan accuracy across the planning benchmark is 58.59%. Strong performance on the component checks does not guarantee that the model will assemble a valid, goal-reaching sequence.
The original error table is useful because it distinguishes two failures that often get collapsed into “the agent got it wrong.”
| Model | BC (%) | GS (%) | Other (%) |
|---|---|---|---|
| o1 | 17.97 | 17.89 | 5.55 |
| o1-mini | 68.91 | 5.94 | 11.95 |
| GPT-4o | 81.56 | 3.59 | 6.09 |
| Claude 3.5 Sonnet | 28.59 | 44.77 | 6.09 |
Original Table 4. BC means breaking a constraint; GS means the plan is legal but does not satisfy the goal. Each percentage uses all 1,280 problems as its denominator, not just failed attempts. Original table.
For o1, constraint-breaking and goal-missing each account for roughly 18% of all tasks. Claude 3.5 Sonnet has fewer constraint failures than GPT-4o, but many more cases in which a legal plan fails to reach the objective. Those require different remedies. A validator can reject an impossible action; it cannot make an otherwise legal sequence useful merely by approving each step.
The business equivalent is a workflow that successfully calls every API but leaves the customer’s actual problem unresolved. Tool-call validity and completion belong in separate checks.
A planning problem can offer only a few choices now while creating many possible states later. Moving one item may free a resource, block a route or change which actions will become available.
The paper compares planning accuracy with measures of action complexity across ten domains. The relationship is stronger with the broader state–action space than with the average number of immediately legal actions.
Original Figure 5, right panel. The horizontal axis is logarithmic. The comparison suggests that the space of future possibilities matters; it does not isolate that factor from every other difference between domains. Source figure.
This gives product teams a better stress test than simply making a prompt longer. Increase the number of dependencies, competing resources and possible intermediate states. A short task with several interacting constraints may be more demanding than a long sequence of independent steps.
The Diplomacy diagnostics show a related gap. Models are asked about units, controlled regions and adjacent locations, then about possible attacks and the support needed to carry them out. The later questions require combining facts into a consequence.
Original Figure 4. Scores summarize the quality of the models’ answers to different kinds of game-state questions. The sharp drop in the final two columns shows the gap between locating facts and using them in a multi-step decision. Source figure.
The chart’s F1 scores are not game win rates. Its value is the contrast between simple and compound questions. An agent may know which resources exist and still fail to work out whether a proposed action can succeed.
Hanabi makes the information problem concrete. Players cooperate to build card sequences, but each sees everyone else’s cards and cannot see their own. They can spend limited hints to tell another player something useful.
The paper’s three-player prompt gives the agent no identifying information about its own cards. It shows another player holding a blue 1, among other cards, while the blue sequence has not yet started. It also records the hints the other player has received and the remaining information tokens.
Seeing the blue 1 is not enough. The agent has to distinguish what it knows from what its teammate knows. Giving a permitted hint may enable the teammate to play safely, but spends a resource that could be needed elsewhere. The paper’s prompt illustrates the decision structure; it is not a demonstrated winning move sequence.
In the reported games, o1’s average score falls from 16.4 with two players to 14.2 with five. Other models vary irregularly with team size. These results do not establish that every additional agent makes performance worse, but they challenge the assumption that more participants automatically improve coordination.
The equivalent test in an enterprise workflow is a handoff where one agent has evidence another lacks. Does the handoff transmit that evidence? Does the receiver know which facts are confirmed, which are assumptions and which actions have already happened?
Diplomacy adds explicit negotiation. In the four-agent comparison, enabling negotiation changes o1’s final supply-center count from 17 to 10. GPT-4o moves in the other direction, from 15 to 17. The effect is not uniform across models.
Even the o1 result is more nuanced than “negotiation breaks planning.” Its reported move and attack success rates improve in that setting while its final position worsens. Better individual actions and a worse overall outcome can coexist. That is precisely why a team should measure the result of the whole workflow rather than infer success from fluent exchanges or accepted proposals.
The expanded settings in the appendix support examining different configurations, but these game experiments are not direct evidence about commercial bargaining or executive decision-making. Some negotiation-quality measures are also assessed by another LLM. Treat them as supplementary analysis, alongside the environment’s actual outcomes.
The SPIN-Bench project provides its benchmark resources and trajectory viewers. Inspecting a complete run is a useful starting point: the initial state, available actions, messages, attempted moves and resulting state reveal much more than a final score.
For formal plans expressed in PDDL, the VAL plan-validation system can check a proposed plan against its domain and problem definition. It is a validator, not a general business-strategy judge. The specification must represent the constraints and goal you actually care about.
For an existing application, apply the same separation with ordinary workflow checks. Record whether each action was valid, whether it produced the intended state change and whether the final objective was reached. Test both a single agent and the proposed multi-agent arrangement on the same cases. Add missing information, a changed resource, an invalid action and a failed handoff deliberately, then inspect recovery.
This complements our coverage of office-agent coordination: selecting the right tool and arguments is one part of success. It also complements SagaLLM’s treatment of multi-agent workflows, where failures and recovery need explicit handling.
A useful acceptance decision is concrete: the multi-agent version must improve goal completion enough to justify extra calls and coordination, while maintaining the required constraint compliance. If it produces more discussion but no better outcomes, simplify the system.
SPIN-Bench is most useful as a challenge to how agents are evaluated. A persuasive plan is not an executed plan. A legal action is not a completed objective. A coherent conversation is not successful collaboration.
Before expanding an agent’s scope, test those capabilities separately and then together. The case for greater autonomy should rest on reliable state changes and recovery under realistic constraints, not on how strategic the model sounds.
SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?, Jianzhu Yao, Kevin Wang and colleagues. arXiv version 1, submitted 16 March 2025. Numerical results and original visuals in this article refer to that version.
Explore how agents preserve state and revise plans after a tool or earlier decision fails.
Explore how predicting action consequences differs from producing a plausible plan.
