How automatic prompt engineering generates and tests instructions, why high scores can mislead, and how to evaluate prompts on your own workflows.
Select a figure to open it at full size.
A team can spend hours refining an AI instruction and still have no reliable evidence that the new wording works better. The useful alternative is to treat prompts as candidates in an experiment: generate several, run them on representative tasks, and keep the ones that improve results on examples they have not seen.
That is the proposition behind Automatic Prompt Engineer, or APE. The research by Yongchao Zhou and colleagues at Toronto, Waterloo and the Vector Institute turns prompt writing into a search process. Its lasting contribution is the evaluation loop. Its most revealing result is that a prompt can score well while describing the wrong task.
This revision replaces the earlier article’s unsupported company examples with an identifiable, tested research method.
APE begins with examples of the desired transformation. For an antonym task, the input might be a word and the output its opposite. A language model receives several such pairs and proposes an instruction that could explain them. Other candidates express the task differently, or infer a different rule.
The system then runs those instructions through the model that will perform the task. It scores the resulting answers against known outputs, eliminates weak candidates and spends more evaluation effort on promising ones. The winner is tested on held-out examples. A persuasive description alone earns nothing; the instruction has to produce useful behavior.
The original method and algorithm also allow the model to paraphrase strong candidates and search again. But repeated refinement brought only marginal gains in the authors’ experiments, so their default method used the simpler initial search. An elaborate self-improvement loop is not automatically better than generating a useful pool and evaluating it properly.
Notice the two distinct responsibilities in the diagram. The proposing model supplies variety; the scoring process decides what deserves to survive. Improving the first cannot compensate for a metric that rewards the wrong behavior.
For the instruction-induction experiments, the authors used five input–output pairs to propose instructions, typically searched 50 candidates, and evaluated performance using InstructGPT. Tasks included antonyms, extracting letters, translation and identifying causes. These were bounded tasks with answers that could be scored, rather than open-ended executive advice.
In the March 2023 paper version, APE matched or exceeded the human-written instructions on all 24 instruction-induction tasks and on 17 of 21 curated BIG-Bench tasks. Across the first set, its interquartile mean performance was 0.810 versus 0.749 for the human instructions. That aggregate reduces the influence of the highest and lowest scores; it does not mean every application improves by the same amount. The comparator was the benchmark’s human instructions, not an unlimited team of expert prompt optimizers.
The experiments used historical models and task collections. They establish that model-generated search can find effective instructions under a defined evaluation. They do not establish that a particular 2023 prompt is optimal for a current model, or that prompting replaces better data, retrieval or training when those are needed.
The paper’s rhyming example deserves more attention than the headline win rate. In Appendix Table 8, an instruction about capitalizing the first letter of a string scored 1.00 in the zero-shot rhyming evaluation. When the model was also given examples, that same instruction scored 0.03. A straightforward rhyming instruction scored 0.55 and 0.61 respectively.
The point is not to adopt the capitalization instruction. It is that optimization can discover a behavior that is effective in one setup without being a clear, portable specification of the intended job. Adding examples changes the context and can destroy the apparent advantage. Prompts therefore need testing in the complete format that will actually be deployed.
A second experiment makes the objective problem explicit. When APE optimized truthfulness alone, it could discover instructions that avoided false statements by declining to answer. The researchers separately measured informativeness because an evasive answer can be truthful and still useless. A customer-support evaluator that rewards only the absence of mistakes could select the same failure pattern.
A useful first application is a bounded workflow such as routing support tickets, extracting a specified field or deciding whether a document contains required evidence. Define the output and assemble examples that distinguish correct behavior from plausible mistakes. Include ambiguous cases and cases where the system should abstain.
For ticket routing, compare a plain human-written instruction with generated alternatives. Score correct destination, unnecessary escalation and incorrect confident routing separately. Freeze the production context—including examples and output format—during the comparison. Keep a final test set out of the search process, then inspect changed decisions rather than accepting an aggregate gain alone.
The acceptance condition should reflect the service: fewer costly routing errors without a surge in escalations or latency. If improvement disappears on a later time period, another team’s tickets or a model update, the prompt has not earned a broad rollout. This is also why guardrail evaluation needs both safety and usability measures.
The authors’ APE repository separates prompt-generation data, evaluation data and templates through its find_prompts interface. It is useful for understanding and reproducing the search structure. Its historical model integrations should be checked before use with a current provider.
A small existing evaluation script may be sufficient: store candidate instructions, call the target model on the same cases, validate outputs and compare scores. Keep generation examples, selection examples and final test examples distinct. Version the model, instruction, examples and scoring rules together; otherwise a rerun is testing a different system without saying so.
Start with one task and a modest candidate set. Record per-case outcomes and total search cost. Promote the winning prompt only after reviewing its meaning and checking that the gain survives the held-out test. Automated search reduces the writing burden; it increases the importance of specifying what success means.
The valuable asset is a repeatable evaluation that can tell a useful instruction from a lucky one. APE makes that distinction concrete. Use generated prompts to widen the search, while keeping human ownership of the objective, unacceptable failures and release decision. A simpler prompt that transfers reliably is often worth more than a spectacular score on a narrow test.
Yongchao Zhou and colleagues, Large Language Models Are Human-Level Prompt Engineers, ICLR 2023; arXiv version 2, March 10, 2023. The workflow, benchmark results and rhyming comparison above refer to this version.
See why a model that ranks answers well may still provide a poor training signal.
See why executable SQL still needs checks against the business question and intended metric.
