Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
TravisMuhlestein 
posted an update 7 days ago
Post
122
AI orchestration > bigger models?

Over the past few months I've been experimenting with a different approach to AI-assisted creation.

Instead of asking one frontier model to do everything, I explored what happens when specialized capabilities are orchestrated into a single production workflow.

The project coordinates drafting, editing, illustration, multilingual translation, narration, publishing outputs, and marketing assets into one end-to-end system.

The first real-world validation is The Isaiah Chronicles, the first release in a planned 10-book series.

Over roughly two months the workflow processed more than 14 billion tokens, with continuous experimentation around evaluation metrics, quality gates, reproducibility, and orchestration strategies.

One observation kept repeating itself:

Improving the orchestration often produced bigger gains than changing the underlying model.

I'm curious whether others building AI systems are seeing the same pattern.

How much of the next leap comes from better models—and how much comes from better orchestration?

🔗 https://theisaiahchronicles.com

Orchestration beat the model in your runs, and the way to know is to freeze one of them.

14 billion tokens over two months is a lot of evidence, but it is evidence about one moving system. The orchestration and the model both changed across it, so the gains got attributed to whichever one you happened to be touching that week.

The cheap test is a held constant. Take one stage you have already shipped, drafting or translation. Keep the graph, the prompts and the quality gates frozen, swap only the model. Then freeze the model and change only the orchestration. Same inputs, same gates, same scorer. Two deltas you can put side by side.

My guess is orchestration still wins, but for a reason that reframes the claim. Orchestration mostly buys retries, checks, and narrower asks. Those pay off in proportion to how often the model is wrong in a way your gate can actually see. That makes the win a property of your gates as much as of the model. A stronger model moves the same number only if the gate can see the errors it still makes.

Which of your quality gates has ever rejected a stage output, and how often does it fire?