measuring code quality in an agentic world

My buddy Scott at Red Hat shared https://redmonk.com/sogrady/2026/02/10/besieged/ in the group chat.

It's a worthwhile summary of this point in development history.

Earlier I was appreciating the zeitgeist that maybe we were near the top of the S-shaped curve of what LLM could do, things might be settling down a bit. But recently I've been catching a different spin, that specifically the model releases last November really pointed to a new level of capability - something beyond the intuitions programmers experimenting with LLM the years before would have developed.  (The future might be the dev as head chef, with a bot sous chef bossing around a small hoard of underling agents.)

Scott mentioned Red Hat is pivoting to agentic workflow: "One interesting thing our CTO said was, we will not measure AI consumption, we will measure code coverage, quality, etc. Focus on outcomes."

Now, casting a blind eye to consumption reminds me of a tale from AOL/Millennial Media, when they thought pivoting to the cloud/AWS was giving them a bunch of dev and qa servers "too cheap to meter" - an expensive lesson they had to correct course around. And of course, there's the environmental impact (energy and water consumption - not to mention the "environment" of the crazy costs of GPUs and memory). And for all the democratization of code LLM offers, the idea that you must shell out month after month on a service (or buy some ridiculously priced hardware) in order to be competitive threatens to undermine the self-starting developer paradigm that got its start when home computers became cheap in the 80s.

But as I try to figure out how I can maintain my role as mediator between Business Folk and Machines I want to think about the second part of Scott's paraphrase, "code coverage, quality, etc"

Too often (in my heretical, anti-reductionist opinion) organizations go to quantify code coverage via unit tests percent coverage. But AI is going to require a new sense of what "quality" is:

* ability for future gens of your AI hoards to pick things up and iterate on earlier work

* consistent behaviors (externally testable)

* human-friendly UX (when applicable)

* edge cases covered

* one layer of AI check looking for weirdass shortcuts AI may have taken

* security, security, security

All of these have parallels in what traditional software development has to be keeping an eye on, but with humans being not as deeply embedded in the core loops, we will need more a more sophisticated way of quantifying how well things are going.

The AI Vampire

 My buddy Scott is a highup Project Manager at Red Hat. We hang out on a tiny group chat.

Scott has a huge experience with Open Source (and did some pioneering work in containerization) and is on board with the idea that AI is the future of development. He's really worried that the good guys get busy, overcome their inertia and reservations, and learn the arts of leveraging AI to make good stuff (with efficient token/energy use) and learning how to defend things from the bad guys, who never seem to have the same kind of compunctions about using any tool they can find.

He's fighting a war on two fronts; he yells at me for bringing up stuff from the AI-skeptic side, both from absolutists who will push back against AI tooth and nail, as well as from middle of the road developers who find utility in these "Jr Programmer Level of companions" but keep bumping their head into the limitations and confusion that results as context windows get filled up and what not, as well as some of the infamous AI gaffes making the rounds.

The other front is against executives who are too gung-ho about it all - who say "just build it" without acknowledging how the problem is no longer the coding - we now are in an age of surplus programming/build potential - but deciding what to build that will provide sustainable value, as well as socializing and building community around what gets built.

One of Scott's favorite articles is "The AI Vampire": 

Here were my takeaways:

* Yegge claims that a real corner was turned last November, in terms of what models like Claude Opus could do vs the earlier stuff. I'm currently nursing a theory that says, there might be a qualitative difference between those "$200/mo" models vs the "$20/mo" models that both the naysayers and the CEO types have been messing with, which has increased the skepticism in the first camp (as they run into limits) but with CEOs empowered enough to build cool stuff and not getting bothered by the MVP/prototype level of what they make. (I don't think Scott agrees with my analysis, but anyway) But Yegge says stuff like the long vaunted "10x Programmer" is now actually unlocked.

* Yegge points out that this AI-driven 10x mode is actually exhausting (hence the title of the piece) I'm guessing coding with AI takes a significant part of the fraction of doing it "by hand". It's a more concentrated set of demands and focus.

* and So we have a classic "where does the value go" situation. If the company tries to get 10x the work all the time, endless go-go-go mode, that's just a recipe for burnout. Conversely, if it's more like employees just put in 1/10 the effort, that's a plan to get swamped by competitors who are increasing their productivity. Yegge thinks its crucial we find a balance for the value capture.

And so I'm working but wondering what the rest of my career looks like - and where I am on the bellcurve of adoption...

watch them AI quotes

 I was really bummed to see journalist Benj Edwards got in trouble with AI fabricating quotes (he claims he was loopy with COVID and mistook paraphrases from one tool for actual quotes from another.)

I really enjoyed his previous work on computer history and retrogames - his occasional appearance on the podcast Retronauts, and I even bought a arcade stick for the Atari 2600 he was making as a side hustle.

Plus, this piece on "10 things I learned from burning myself out with AI coding agents" was right on - from the sheer fun of it, to the challenges when an AI reaches the end of its competency and familiarity...

&&

 Avoid unsafe "&&" Operator for Conditional Rendering in React and Next.js

Basically the TLDR is:
isVisible && <item />
to control visibility in React isn't great... edges cases of falsiness (like "NaN") may show up when you were hoping for a blank. So use the ternary operator with "null" as the false case instead.

On my team I pushed it a little further and asked if we could prefer 

isVisible ? <item /> : null;

over

isHidden ? null : <item />

admittedly the case the triggered me was like

isReadOnly || !featureFlagForItem ? null : <item />

which is kind of a tangle to read.

a good url scheme is a joy forever

Unsung heroes: Flickr’s URLs scheme

I really do love a good URL scheme. The first time I was blown away by an elegant scheme was Jira, back around 2010 I think. They incorporated ticket numbers directly in the path, and I had a brief flicker of "wait, they can do that now?" (at that point, URL routers weren't ubiquitous, and URLs often had to be implemented as a physical file structure.) 

URLs remain such a powerful form of UI, even after the industry's pivot to "mobile (app) first". Even to this day, people always aren't in a hurry to download your company's app, and a lovely URL can be so expressive - so much kinder to humans than "robot throwup" QR codes.

(And don't get me started on ?fbclid privacy-breaking tracking nonsense; I wouldn't be QUITE so resentful if they weren't so long and winding - no GUID needs to be that long.) 

a shinier net

So, scammers are using AI to make high-grade, legit looking sites. 

Most likely this is a ruse (with fake books and authors) to sucker would-be authors in to pay for "Resources for Writers" etc, or to just harvest contacts.

As KJ Charles puts it:

"I've just realized. Banks used to be big imposing high street buildings because it gave people confidence they had money and weren't fly by night. A big elaborate website was the internet equivalent: proving someone had invested £ and was here to stay.

You can't trust that any more."

It reminds me of "Nigerian Prince" scams... people wondered why they were so transparently false and full of typos, but the smart view was the clumsiness and blatancy was a feature, not a bug - they were casting a very wide net and wanted only the most gullible fish. 

This changes that equation, but only somewhat. It's still pretty obvious it's a fake site (you can google based on the fake titles and author names if you want) since everything is put behind a "contact us" personal data harvesting form, but that is lurking beneath a very polished veneer. 

ai-yi-yi

Like everyone else, still trying to get a read on the present and future of AI in tech. Most everyone is daunted by being on the wrong side of a "have and have not" divide in terms of being able to use these tools, and thoughtful people have concerns ranging from the cultural to the environmental about the long term impacts.

People's results when handing parts of development over to AI seem to vary greatly, and it's hard to know if people are tackling different problems (so it's not apples to apples comparisons) or if some people are using better tools (or throwing more tokens at things!) or using methods to provide more structure to the AI, vs the "prompt and pray" approach.

It may be useful to see places that are taking more experimental and data-driven approaches, than just relying on anecdotes...

One paper was "How AI Impacts Skill Formation" (by some Anthropic fellows): 

from the abstract:

"Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. [...] We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library."

Another video is "We Studied 150 Developers Using AI" that had a special focus on longer term maintainability, vs just the initial cost.

"Well, the headline is there was no significant difference between the cost of maintenance between AI and human generated code. That's interesting and perhaps not what we might have expected. Code written with AI assistance was no harder to change, no easier to change, no worse in quality and no better in quality."

And also, besides some of the speed up in dev, dev who used AI on the the regular tended to make more idiomatic "normal" code.

Anecdotally this lines up with some of my experiments with Claude Code - where you give high level direction as a PM/QA rather than using CoPilot or ChatGPT has a coding buddy. I feel like the sweet spot is using AI to augment rather than replace the classic developer role. Maybe some of that's wishful thinking, maybe some of that's recognizing the cost these systems have and how they really can lose the thread sometimes. (I mean so can humans, but it has a different feel...)