Agent Optimization
Most optimization work changes instructions and model selection by hand, without the measurement required to confirm that quality or cost moved.
The evidence needed to confirm an improvement is usually absent, which leaves quality unverified and cost unattributed. This engagement builds that measurement first, records the current configuration against it, then optimizes each layer on results rather than on judgement.
- Baseline: what you run today
- Context: what reaches the model
- Harness: the loop around it
- Token and cost: routing, right-sizing
- Scored run: quality, cost, latency
- Held-out gate: adopt or roll back
Baseline first
Nothing is tuned until the current configuration is measured.
Gated
A candidate ships only if it keeps the work you already solve.
Reusable
The measurement stays with your team after we close.
Where it applies
An agent already in production, where spend has climbed and quality moves run to run with no attributable cause.
What we measure
Quality, cost and latency on one scored task set, recorded before and after every change.
What you keep
The task set, the scoring and the runner, on open source your team reruns without us.
Fit
When this applies
An agent already in use, where quality swings, spend has climbed, and nobody can attribute either to a specific change.
Live systems
Quality varies run to run with no clear cause
Rising cost
Token spend growing faster than usage
Plateaued quality
Prompt changes stopped producing gains
Large repositories
Code beyond any single context window
Scope
Where agent optimization goes wrong, and how we address each
Most teams recognise several of these from the past quarter. Every one is measurable, and the tooling behind each response is open source we maintain and you keep.
The five layers
Most teams change the first and the last, because both can be edited by hand. The gain is usually in the three between them.
- 01InstructionsGEPA and the DSPy optimizers
- 02Contextranking, summarization, budget
- 03Retrievalscored on what is read
- 04Harnessthe loop around the model
- 05Modelassignment per role
| The problem | How we address it |
|---|---|
| Model swapped, bill up, quality unproven | The current configuration is scored in SuperOptiX before anything is altered, with cost and latency recorded beside quality. The baseline is a rerunnable artifact rather than a report. |
| Prompts tuned by hand, never scored | Instructions are optimized with GEPA, which runs across all eight runtimes SuperOptiX supports, alongside the DSPy optimizer family where the task suits it: BootstrapFewShot, COPRO, BetterTogether and Ensemble. |
| Every step routed to the frontier model | Steps are scored individually and model assignment is set per role. TurboAgents handles right-sizing and quantization on token-heavy paths. |
| Tracing in place, evaluation missing | Existing traces become the scored task set, and the resulting decision is written to an Agent Quality Record that remains auditable afterwards. |
| Evaluation overlaps the tuning data | Held-out tasks are sealed and the manifest is fingerprinted, so a score reflects work the optimizer never saw. SuperQode enforces the same split for harness work. |
| Model and harness treated as one decision | Each is measured independently on one task set. MetaHarness optimizes the harness, SuperOptiX optimizes the agent, and the record shows which produced the gain. |
| Retrieval left at defaults | The GEPA RAG adapter optimizes retrieval against your vector store, scored on what the agent reads rather than on retrieval metrics alone. |
| Tool descriptions written once, never tuned | The GEPA MCP adapter optimizes tool descriptions and selection against the same task set, over stdio, SSE or streamable HTTP. |
| Context and memory grow unmanaged | Ranking, summarization and context optimization are tuned as one layer, with the window budget derived from measurement. |
| Gains adopted without a regression check | Every candidate is scored against the existing solved set, and a regression blocks adoption. |
| Optimization cannot be repeated in-house | Every component is open source we maintain. The task set, the scoring and the runner are handed over with a documented routine for rerunning them. |
Process
How we work
Build the measurement
Tasks drawn from your workload and the failures you keep hitting, assembled into a scored set with a held-out split sealed from anything that tunes the agent.
Establish a baseline
The current configuration scored across that set in SuperOptiX, with cost and latency recorded next to quality so trade-offs stay visible.
Optimize each layer
GEPA and the DSPy optimizers on instructions, the RAG adapter on retrieval, the MCP adapter on the tool surface, MetaHarness on the harness, and model assignment set per role.
Gate and hand over
A candidate is adopted only when it beats the baseline without regressing solved work. The decision is recorded, and the runner stays with your team.
Deliverables
What you receive
Scope is set from your requirements rather than from a fixed package. Whatever is agreed at the outset is built, measured against your own tasks and handed over, and every component runs on open source your team keeps.
Evaluation set
Scored tasks from your workload, delivered to your team to keep
Optimized configuration
Context, retrieval and harness tuned, with the runs behind each decision
Routing plan
Which model handles which role, chosen on measured outcome
Cost position
Spend measured before and after, with the quality trade-off stated
Scope
Engagement Options
Three columns, scoped and bought independently. Most engagements open with Context, where the quality problem and the cost problem usually share a root. Scope and cost are quoted against your systems once we have seen them, since the size of the task set follows the number of agents and the shape of the failures you are hitting.
Typical starting point
Context
What reaches the model
Typically 3 to 5 weeks
Context and retrieval
Most of the quality and cost sits in what gets assembled for each call.
- Context assembly reviewed and tuned
- Retrieval quality improvement
- Instruction and skill optimization
- Compaction and window management
- Held-out gate before adoption
Harness
The loop around the model
Typically 4 to 6 weeks
Harness and tooling
Tools, permissions, checks and control flow, tuned as one executable artifact.
- Harness measured across variants
- Tool surface and permissions tuned
- Recursive routing for large repositories
- Evaluation set you keep
- Documented rollback path
Token & Cost
Spend under a quality floor
Typically 3 to 5 weeks
Routing and right-sizing
Frontier models often run work a smaller model handles at the same score.
- Token and spend analysis
- Model routing per role
- Local and open model evaluation
- Quantization where accuracy allows
- Cost measured before and after
Open source
Underlying systems
These are our systems, published openly. Engagements tune whatever you have in place.
MetaHarness
Optimizes the executable harness around a coding agent.
DSPy Code
CLI for building and optimizing DSPy programs with GEPA.
RLM Code
Recursive language models over oversized codebases.
CodexOpt
Optimises AGENTS.md and skills with GEPA.
TurboAgents
Quantization and right-sizing for token-heavy agents.
SuperQode
Harness benchmarking with held-out gates.
Questions
Questions
Discuss Agent Optimization
Send us your current configuration. Stop after the baseline and the measurement is still yours.
London and San Francisco. Remote or on site.