Wednesday, April 3, 2019

unpopular opinion: 'developers do their own testing' doesn't scale

Almost two decades ago, I ran into Mr. Bad's Things to Say When You're Losing a Technical Argument. My favorite was the shortest, and first on the list:
1. That won't scale
So, keep that in your back pocket.

Sometime over the past decade, the concept of "We don't need QA as a separate discipline - we can trust our developers, fortified by automated testing".

Almost exactly five years ago I wrote a blog entry about joel on software and the quixotic nature of complete testing . I still stand by most of what I wrote there - especially about how hard it is to get coders to bruise their own ego, and think outside the box.

But the "No QA" policy is so hip and trendy in tech - and yet I know of at least two companies following that path that are really struggling with frequent emergencies. And of course, it's pretty smart and productive people espousing it.

My current theory is that the idea that developers know enough about the holistic system to do a good job of knowing what will break - not just coding to prevent it, but then taking the next step of meaningful tests well "outside the box" of where they've been codin - only works with a limited scale code base. As a team grows and codebase ages:
A. you probably have a smaller proportion of "rock stars" than you had early on
B.  the newer people haven't witnessed the growth from the simple core to whatever gnarly forest you now have, knowledge of which helps in intuition and sleuthing
and
C. just the number of connections is increasing, exponentially or at least geometrically. There's just more there there! Maybe- just maybe- "developers do their own testing" doesn't scale across time and space.

This is why my favorite work setups involved a small group of dedicated QA folk. They were engaged enough to ask good questions (different from that "oh we'll let offshore QA bang on it overnight" I've seen elsewhere), smart enough about the codebase - and at a high enough level - to have a gut feeling for where a new change might be problematic. It's a bit adversarial (but in a congenial style) in a way that a coder finding their own bugs isn't, even if the coder has a reputation to protect.

No comments:

Post a Comment