When Traditional Testing Was No Longer Enough
I sometimes recall (with a touch of nostalgia) 2004, the year we started working with automated testing. Back then, the digital landscape was very different: the tools were rudimentary and complicated to set up, each test required time and meticulous attention to detail when interacting with identifiers, and getting everything to work in the CI/CD environment was an act of faith. If you dealt with Selenium back then, you know what I'm talking about. But in return, we had the satisfaction of testing faster and "earlier," freeing up manual time for more relevant tests. That was our responsibility. Fast forward to 2026, and considering artificial intelligence in the form of bots, agents, and code generators, I often wonder: who is responsible for testing them? Who ensures that they "work" correctly? Today, we talk without batting an eye about assistants that understand, reason, make decisions, call APIs, or coordinate with other systems. But amidst all this technological hype, there's a detail that sometimes goes unnoticed: these agents aren't and don't work like traditional software. And yet, they still need what has been Redsauce's DNA since our beginnings: quality.
The Day an AI Made Things Up
You probably remember the first time an AI gave you a completely nonsensical answer. In my case, it wasn't a bug in the traditional sense, the kind you reported with a scenario, preconditions, steps to reproduce, and boom! there it was. There was no reproducibility here; it simply decided… to make things up. Not to lie, because that wasn't its intention (I hope), but rather to fill in the gaps with something that sounded believable if you didn't know what it was talking about. At that moment, you realize that everything has changed, that the challenge is no longer fixing lines of code, but ensuring behavior and that the agent: Is consistent, useful, follows the rules, and is honest about its limits. Doesn't deviate. Doesn't cheat. Doesn't hallucinate. Is safe. Is unbiased. Is not toxic. ...and that it always stays that way. We definitely needed a new kind of testing.
From Functional Testing to Criterion Testing
For years, our QA suites had verified clear conditions: if you do this, then that should happen. Cause and effect. Input and output. But with agents, everything changes. The correct answer isn't always unique, it is not deterministic. The difference is sometimes a range, sometimes a nuance, sometimes avoiding a temptation (like fabricating data). And other times it's following the defined rules, from the tone to the length of the response. So, once again, it was time to reinvent ourselves. It was no longer just about verifying that a function responded; it was about validating that it responded appropriately. With accuracy, consistency, and responsibility. With all this in mind, we set to work designing test suites that mixed prompts, conversational scenarios, limits, intentional traps, contradictory instructions, stress tests... The goal was to understand not only the agent's optimal behavior, but also how it reacted when queries deviated from the script, where it hesitated, where it failed, or where it was exceptionally creative.
Are we facing a new profession?
Clearly, given that AI in any of its forms is a relatively new "product," the ways of testing it must necessarily be different from traditional methods. But, in essence, we are doing what we have always done: ensuring the quality of a product. Only now, quality is not measured in page objects or functionalities, but in behaviors, criteria, and decisions. AI agents don't come with instruction manuals, and none guarantee correct answers. I refer you to the disclaimers found within them:
But can (or should) we accept these potentially erratic responses as if AI were an unfathomable mystery? Not that it isn't, and increasingly so. But the point is to ensure, as far as possible, that within its range of responses, these are coherent and adhere to certain principles. A challenge similar to the one we faced in 2004… and at breakneck speed.
And the final chapter
Leveraging the experience gained from combining tools, specific prompts, results analysis, and evaluation frameworks for AI agents, at Redsauce we've built a methodology that allows us to:
- Validate accuracy, consistency, and accountability.
- Detect fabrications, deviations, rule violations, hallucinations…
- Test variants, compare models, and measure regressions.
- Integrate all of this into CI/CD processes.
Essentially, we're applying the same obsession with quality to the world of AI that has guided us for the last two decades. Whether you're starting to work with agents, already have one in production, or simply want to ensure your systems respond as they should, I'd love to hear from you. This is exciting new territory, and we'd like to explore it with you. Write to us at info@redsauce.net or use the contact form and we'll be happy to reply personally. Let's talk!