Many technology leaders I speak with are caught in a peculiar loop: using AI systems to evaluate other AI systems… which are in turn evaluated by other AI systems. It’s battle bots all the way down.
The problem isn’t the evaluation—it’s the endless recursion.
I recently worked with a CTO who had implemented three separate AI evaluation frameworks to ensure their generative AI outputs were “high quality.” Each framework generated its own reports and metrics, which then required another system to reconcile the differences.
The result? A team of engineers spending more time managing AI evaluation systems than solving actual business problems.
What they really needed wasn’t more sophisticated code—it was clarity on what “quality” actually meant for their specific business context.
When I asked what success looked like from a business perspective, the room fell silent. Nobody had translated the actual business needs into evaluation criteria before building the systems.
Quality AI isn’t determined by how many layers of evaluation you add. It’s determined by how clearly you’ve defined what matters to your specific context.
Try this instead:
- Define no more than three business outcomes that matter most
- Identify simple, direct measurements for those outcomes
- Build one lightweight system to track those measurements
One client reduced their AI evaluation stack from six systems to one and saw both better outcomes and 70% less engineering time spent on maintenance.
In the battle bot arena of AI tools, the winners aren’t the most complex machines—they’re the ones most precisely aligned with their purpose.
What’s one evaluation metric you could simplify or eliminate this week?
Christopher Grant
Founder, Nebari Consulting
Need clarity on a specific tech challenge? Reply to this email, and let’s talk.
Spotted a typo? Consider it a feature not a bug. Now you know I’m not an AI 🤖