Every Team's Code Looks Good Now. Here's What to Check Instead

A client came to us with one small change. Working through it, we found that delivering it properly meant rearchitecting half the system, and delivering it the fast way meant living with the consequences for years. An AI assistant would have started writing code immediately. We started a conversation instead.

That conversation is the thing you are actually buying when you hire an engineering team, and it has become the hardest thing to assess. Stack Overflow's 2025 Developer Survey of more than 49,000 developers found that 84% use or plan to use AI tools in their development process, yet only 33% trust the accuracy of their output, and 46% actively distrust it. Every team ships clean-looking code now. Clean-looking code stopped being a signal.

The old evaluation methods assumed that output quality reflected team quality. That assumption no longer holds, and most vendor checklists have not caught up. What follows is a set of tests that produce evidence rather than claims, whether you are shortlisting an agency or auditing the team you already have.

Key Takeaways

• AI-generated code is idiomatic and consistently styled even when the logic is wrong, which removes the surface signals reviewers used to rely on.

• Google's DORA research found AI acts as an amplifier, magnifying the strengths of high-performing organizations and the dysfunctions of struggling ones. The team's underlying system matters more than its tooling.

• Developer self-reports have become unreliable evidence. A two-year JetBrains telemetry study found large behavioral changes that half the developers involved did not notice in themselves.

• Ask how a team's engineers use AI rather than whether they use it. Anthropic's randomized trial found the interaction pattern, not the tool, predicted comprehension.

• Verifiable external credentials and multi-year retention are two of the few quality signals that do not originate in a vendor's own marketing.

• A short paid engagement with a defined milestone produces better evidence than any reference call.

Why engineering quality got harder to judge

AI-generated code fails in ways that look like competence, which is why reading a code sample tells you less than it did two years ago.

Before AI assistance was standard, weak code announced itself. Naming was awkward, style was inconsistent, shortcuts were visible. A reviewer could scan a pull request and form a fast impression of whether the author understood the problem. Generated code removes those tells. It is idiomatic, consistently formatted, and structurally tidy regardless of whether the logic underneath is right.

The failures moved somewhere harder to see: misread requirements, edge cases handled plausibly but incorrectly, logic that solves a near neighbor of the actual problem. Stack Overflow's respondents named this directly. The single biggest frustration, cited by 66% of developers, is AI output that is almost right but not quite, and 45% say debugging generated code takes longer than writing it themselves.

The volume made it worse. Faros AI's 2026 engineering report, built on two years of telemetry from 22,000 developers across 4,000 teams, found that under high AI adoption, bugs per pull request rose 54%, median time in review rose 441.5%, and 31.3% more pull requests were merged with no review at all. Review capacity did not scale with generation capacity, so the gate that used to catch these problems is now propped open on a lot of teams.

None of this means AI-assisted teams produce worse products. It means the artifact you used to inspect no longer carries the information you need. We see the downstream version of this regularly in vibe coding audits, where a working application turns out to have eight to fourteen structural findings on a first pass.

AI amplifies the team you already have

The most useful research finding for a buyer is that AI does not change what a team is. It makes the team more of what it already was.

Google's 2025 DORA report, drawn from nearly 5,000 technology professionals, put it plainly: AI's primary role in software development is that of an amplifier, magnifying the strengths of high-performing organizations and the dysfunctions of struggling ones. Adoption reached 90%, a 14% rise year over year, so tooling is now table stakes. Two teams with identical stacks and identical subscriptions will produce very different outcomes with them.

This reframes the entire vendor conversation. Questions about which models a partner uses, or what percentage of their code is AI-authored, sort candidates on a variable that barely predicts anything. The variable that predicts something is the system around the tools: how work gets specified, who reviews what, and what happens when someone disagrees with the output.

Why a team's own account of its speed is not evidence

A vendor telling you that AI made them 40% faster is reporting a perception, and perception has measurably decoupled from behavior.

JetBrains' Human-AI Experience team ran the study that shows this most clearly, presented at ICSE 2026. Researchers analyzed two years of IDE telemetry from 800 developers, 400 who used an AI assistant and 400 who did not, covering roughly 152 million logged events. AI users increased their monthly delete and undo actions by about 100. Non-users increased theirs by about 7.

Then they asked the same developers what had changed. Half reported no change in their editing behavior at all. The rework was real, sustained, and invisible to the people doing it.

The pattern repeats elsewhere. METR's randomized controlled trial put 16 experienced open-source developers through 246 real tasks in their own repositories and found they took 19% longer with AI tools, while estimating afterward that AI had made them 20% faster. METR has since flagged that result as historical, noting in early 2026 that selection effects made its follow-up hard to interpret, so treat the specific number carefully. The perception gap it documented is the durable part, and it is consistent with what JetBrains measured independently.

For a buyer, the practical consequence is narrow and useful. Discount self-reported productivity claims in a pitch. Ask instead for something observable: cycle time on a comparable project, the share of pull requests merged without human review, incident counts before and after adoption. A team that measures itself can answer. A team that does not will change the subject.

5 tests you can run in a first call

Replace questions a partner can rehearse with exercises that produce something you can judge. Each of these takes under an hour and works in a first or second call.

Give them a small feature that isn't small

Bring a real request from your backlog that looks trivial and has a hidden dependency you already know about. Ask how they would approach it and what it would take.

What you are watching for is whether they find the dependency before committing to an estimate, and what they do once they find it. Strong answers name the coupling, price both the fast path and the correct path, and sometimes recommend not building it yet. Weak answers give you a number.

Ask what they would reverse

Ask for a decision they made on a past project that they later regretted, and what it cost to live with or unwind.

Rehearsed answers to this question have no price attached. Real ones do, because the person has paid it. The specificity of the cost is the signal, not the eloquence of the reflection.

Ask how AI output gets reviewed, then ask for the number

Every vendor now has a fluent answer about their AI review process. Follow it with a measurement question: what share of your pull requests merge without a human reviewer, and how do you know?

Given the 31.3% rise in unreviewed merges in the Faros dataset, this is a real risk and not a hypothetical one. A team with review discipline has instrumented it. A team without one will tell you it never happens.

Ask how their engineers use AI, not whether

Anthropic researchers Judy Hanwen Shen and Alex Tamkin ran a randomized trial with 52 professional and freelance developers learning an unfamiliar Python library, published in January 2026 as How AI Impacts Skill Formation. Participants who used an AI assistant scored 17% lower on a comprehension quiz taken without AI, which the authors describe as roughly two grade points. The largest gap appeared in the debugging questions.

The more useful finding sits underneath that headline. Outcomes tracked the interaction pattern rather than the tool. The researchers sorted participant behavior into six patterns and found three that preserved learning, scoring between 65% and 86%, against three that did not, scoring between 24% and 39%. The pattern that scored highest was among participants who asked only conceptual questions and corrected their own errors. The fastest group delegated code generation wholesale and finished with the second-lowest comprehension score in the study.

So the question worth asking a partner is what their engineers ask the model, and whether anyone on the team can explain a generated module without reopening the chat history.

Ask who will understand this system in month 30

Most evaluation focuses on the first release. The expensive problems arrive later, when the person who made the decision has left, and nobody has written down why.

Ask about tenure on their longest engagement and what handover looks like when someone rotates off. We at Softonix have been the frontend team on Wawibox for more than 7 years, and on DriveHRIS we rebuilt over 100 screens across 5 years into a platform now serving more than 10,000 daily users across Canada. These numbers aren’t achievements in themselves. They are the reason someone on our side can still explain why a2021 decision1 was made that way.

What an external skills signal looks like

A verifiable credential is one of the few quality signals in a vendor evaluation that does not originate in the vendor's own marketing, which is precisely what makes it worth checking.

Every agency website claims senior engineers. The claim is unfalsifiable as written, and buyers have learned to discount it. A certification issued by a third party and published with a public verification page is different in kind: you can open it yourself and see a name, a level, and a date.

At Softonix, passing the official Vue.js Senior Developer certification is mandatory for our frontend engineers. More than ten of them hold it, and mine is here if you want to see what the verification page looks like. The exam was developed under the review of Evan You, the creator of Vue, together with members of the Vue.js core team, and it combines multiple-choice questions with practical coding exercises. It is not a framework quiz. It covers performance, testing, security, and web vitals, which is to say, it tests whether an engineer understands what is happening under the hood of the framework rather than which API to reach for.

Certificates.dev now runs a team program built around this exact problem, letting organizations standardize what junior, mid, and senior mean across squads, rather than leaving it to each hiring manager's judgment.

The honest limit matters here too. A certification proves depth in a technology. It does not prove architectural judgment, and it does not prove that someone will tell you a feature is a bad idea. Treat it as a floor you can verify, then run the five tests above for the things no exam measures. 

If you want to talk through how this applies to a team you are evaluating, book a call.

The signals that stopped mattering, and what replaced them

Several standard evaluation criteria have quietly lost most of their predictive power over the last two years, while a few new ones have become far more useful.

How we would want to be evaluated

The strongest evidence about an engineering team comes from working with them briefly, which is why we structure the start of an engagement as a test rather than a pitch.

We agree on a first milestone and deliver it in 2 weeks. If you do not see the value, you do not pay. That arrangement exists because the alternative, asking you to choose based on case studies and a call, requires you to trust exactly the kind of self-report this article argues against. Two weeks of real work in your repository against your definition of done produces better information than any proposal document.

It also tends to surface the scoping question early. On Autobound we started with a single engineer and grew into the full engineering function behind the web app, AI engine, and Chrome extension, through a $4M seed round and a fivefold increase in development velocity. That relationship started small, and the early work determined whether the judgment was there.

Wrapping up

The teams worth hiring in 2026 are not the ones with the best AI tooling, because that distinction has evaporated. They are the ones who can tell you why the code is shaped the way it is, who catch the small feature that is not small, and who have kept enough people on a system long enough that somebody still remembers the reasoning.

None of that shows up in a code sample anymore. It shows up when you give a team a real problem and watch what they do with the ambiguity in it. 

If you are working through a shortlist or want a second opinion on a team you already have, get in touch, and we will talk specifics.

FAQs

What should I ask a development partner about their use of AI?

Skip the question of whether they use it, since DORA's 2025 research put adoption at 90% among technology professionals. Ask what share of pull requests merge without human review, how they specify work before an agent touches it, and whether an engineer can explain a generated module without reopening the chat history. Teams with real discipline have measured these things and can give you numbers.

Are developer certifications worth anything when AI can pass the exams?

They are worth something narrow and real. A credential like the Vue.js Senior Developer certification, which pairs multiple choice with practical coding exercises reviewed by the framework's creator, gives you a third-party skill signal with a public verification page you can open yourself. It proves technical depth, not architectural judgment, so treat it as a verifiable floor rather than a full assessment.

How do I check a team's code quality before signing a contract?

Reading a code sample has become weak evidence, because AI-generated code is consistently styled whether or not the logic holds. Ask instead for a walkthrough of a past decision the team would now reverse, and for delivery metrics such as cycle time and change failure rate on a comparable project. A two-week paid milestone against your own definition of done gives you more than either.

Should I avoid partners who use AI to write code?

No, and you would have almost nobody left to choose from. The distinction that matters is what surrounds the tool: 31.3% more pull requests merged with no review under high AI adoption in Faros's 2026 dataset, which is a process failure rather than a tooling one. Choose on review discipline and specification quality, not on abstinence.

How long does it take to tell whether a development team is good?

Roughly 2 weeks of real work, which is why we structure our first milestone that way. Reference calls and proposals can be prepared in advance; a scoped delivery against your codebase and your acceptance criteria cannot. The signals appear fast, usually in how the team handles the first ambiguity they hit.

What does a paid trial cost compared with a full evaluation?

A 2-week first milestone is typically a small fraction of a full engagement, and in our case it carries no cost if you are not satisfied with the result. Weighed against the alternative, a mis-hired partner on a multi-month build, the arithmetic is straightforward. The expensive failure is the one you discover in month 6.