RECSIP improves answered-case correctness by preserving model disagreement. Examine its coverage tradeoff, shared mistakes and practical evaluation.
Figures open at full size. Wide tables scroll sideways.
Asking a second AI model to check the first sounds like a straightforward way to improve reliability. But agreement has two very different explanations: the models have found the right answer, or they share the same mistake.
RECSIP, a research method from the Technical University of Munich, makes that distinction worth examining. It asks several models independently, groups similar answers, and gives them another opportunity to choose between the competing responses. When disagreement persists, it reports that disagreement instead of forcing a single answer.
The useful result is a tradeoff. In the researchers’ multiple-choice evaluation, about 83.5% of the answers RECSIP returned were correct. It declined to settle on an answer for about 28.4% of the questions. That could be valuable in a workflow with a good escalation path. It is much less useful if an unanswered question simply becomes an invisible backlog.
RECSIP does not bring in an external source of truth. Its contribution is a procedure for deciding whether the available models converge, and for preserving disagreement when they do not.
The researchers’ workflow has five meaningful steps:
Original Figure 1 from Schamschurko and colleagues. The lower branch matters as much as the loop: unresolved disagreement is an explicit outcome. Full research figure, page 4.
Suppose three models produce two different answers to a question. Two answers are grouped together; the third stands alone. The callback presents a representative of each group. If all three models now select the same option, RECSIP can return it. If two groups remain, the method can stop and expose the disagreement.
That is an illustration of the algorithm, not evidence that the majority was right. The callback gives models an opportunity to reconsider the available candidates. It does not independently validate those candidates, and it cannot select a correct answer that none of the original models supplied.
The grouping step is also more consequential than it looks. The paper considers comparisons based on overlapping words, a model that scores pairs of answers, and extraction of the selected multiple-choice option. Two sentences can sound similar while differing on a decisive detail. Conversely, different wording can express the same answer. A grouping error can make disagreement disappear before anyone has resolved it.
The evaluation uses MMLU-Pro, a benchmark of questions with ten answer choices. The tested combination consists of historical versions of GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro. This tests choosing a known answer from a fixed set; it does not establish equivalent reliability for open-ended legal advice, generated code or a long business report.
The paper reports overall precision of 0.835 for RECSIP, compared with 0.776 for the strongest individual model in that experiment. The essential qualification is in the next table: RECSIP’s precision counts the questions on which it actually supplies a final answer. Its disagreement cases are a separate output.
The original answer distribution makes the operating tradeoff visible.
Original Table 3. “Disunity” means the models did not reach agreement. The overall result includes 3,377 such cases; they have not been converted into correct answers. Source, page 8.
Across the table’s 11,883 questions, RECSIP returns 7,101 correct answers and 1,405 wrong ones. Another 3,377 remain unresolved. Calculated from those counts, it answers approximately 71.6% of the workload, with approximately 83.5% correctness among those answers.
For an operator, both numbers belong on the same dashboard. Looking only at returned-answer correctness rewards a system for refusing difficult work. Looking only at completion rewards it for guessing. You need to know how many errors remain in the completed work and where the unfinished work goes.
The burden also varies by subject. The reported disagreement share is 47% in law and 19% in biology. A single overall result therefore cannot tell a team how large its review queue will be. A workflow concentrated in one domain may behave very differently from the benchmark average.
The paper’s most useful diagnostic is its analysis of 45 incorrect biology answers. In 32 cases, all three models were wrong. In seven, the similarity comparison incorrectly treated responses as agreeing. In four, a model that initially had the correct answer changed to a wrong one during reconsideration. The remaining two cases involved a formatting problem and an answer the authors considered a benchmark error.
Original Figure 2. Shared wrong answers dominate this subject-specific error analysis. The chart explains why agreement cannot serve as proof of correctness. Source, page 8.
These failures point to different fixes. A better answer parser may reduce formatting errors. Better similarity tests may prevent incorrect merging. Neither solves the largest category: all models starting from the same wrong answer.
The formatting example is particularly practical. A model included an answer to a demonstration question before answering the actual question. The extraction logic picked up the earlier answer. The system then compared the wrong piece of text. Before adding a sophisticated debate loop, a team needs to verify that it is comparing the intended answers at all.
There is also a cost consequence. Independent calls and repeated callbacks add model usage and potentially delay. The paper’s correctness results do not establish that this is as fast or as cheap as a single call. A production evaluation needs to measure the whole interaction, including the cases that end without a final answer.
A plausible starting point is a bounded internal classification workflow: selecting a support category, mapping an entry to an approved taxonomy, or choosing which documented policy applies. Each decision should have a finite set of permissible outputs and a reliable way to check historical answers.
For example, a support team could run its existing classifier alongside a multi-model version on a labeled set of past tickets. The multi-model system would return either an agreed category or the competing categories for review. The team would then compare wrong automatic assignments, the share of tickets completed automatically, reviewer workload and total cost.
The adoption question is whether fewer wrong assignments justify the extra unresolved cases and model calls. If the disagreements mostly concern genuinely ambiguous tickets, the queue may be useful. If the system regularly disagrees about routine cases, the review burden may outweigh the improvement. If models confidently agree on the same harmful mistakes, the consensus layer is not addressing the real problem.
For questions requiring external facts, add a separate check against those facts. Multiple models repeating the same unsupported claim is not a substitute for evidence. Our agentic-LLM discussion explains why access to tools and feedback changes what a system can verify; our guardrails analysis covers a different problem, checking whether requests or responses violate a policy.
An existing application can implement the essential experiment without a new agent platform: parallel model calls, stored responses, a grouping function, a bounded callback loop and an explicit unresolved result. For a multiple-choice task, begin by comparing validated option identifiers. Reject malformed outputs rather than silently extracting whichever letter appears first.
Open-ended answers need a more demanding grouping stage. Sentence Transformers’ CrossEncoder interface can score pairs of texts and is one possible component. It is not a truth checker. Test it on pairs that differ in negation, quantities, dates and other details that would change the decision. Choose its agreement threshold using labeled pairs from the intended task.
Keep the first model’s answer as a baseline, retain every intermediate response, and cap rounds and spending. Review a sample of agreements as well as disagreements; otherwise the largest failure category can remain hidden. A useful acceptance rule specifies the maximum wrong automatic-decision rate, minimum useful coverage and affordable review burden together.
RECSIP is interesting because it makes failure to agree visible. Its strongest deployment case is a bounded task where abstention is acceptable and unresolved work has an owner.
We would evaluate it as a selective answering system, with coverage and cost alongside correctness. We would not treat consensus as an independent guarantee of truth. If a simpler single-model threshold or a direct source check achieves the same error reduction with less complexity, that is the stronger operating choice.
RECSIP: REpeated Clustering of Scores Improving the Precision, André Schamschurko, Nenad Petrovic and Alois Christian Knoll. arXiv version 1, submitted 15 March 2025. The figures and table above are originals from that version; percentage calculations from the answer counts are identified in the text.
Explore how confidence scores can support escalation or abstention instead of automatic acceptance.
Compare agent-team structures and the point where adding agents stops helping.
