Source item
Open original source ↗Quoted context
These are the quoted author’s words.
It makes sense to optimize model routing at the harness layer instead of the gateway layer if you want to hillclimb on accuracy/cost for any e2e task. Every task is solved by a combination of a model mixture and agent harness. Every task requires a different mixture of models (+harness logic) to be at the pareto frontier of accuracy and cost. * If you only optimize the model mixture at the gateway layer, you lose the broader context encoded in the harness and only optimize at the LLM completion layer. * If you optimize the model mixture at the harness layer, you can make the "optimal" model choices a priori and while in the agent loop For any given task, the model and harness are probably co-optimized together, so that the model mixture can only exist with this specific harness shape and vice versa
jerryjliu0 ↗