Wednesday, June 17, 2020

not the hill I'm willing to die on but - yeesh, unit tests

Sometimes in my recent round of job interviews, I would let it be known that I have an unpopular opinion- that unit tests are wildly overrated.  Based on the reaction I get from people -- and some biased googling I do -- it feels like I've stumbled into a bit of an Emperor's New Clothes situation - many coders seem to agree with my sentiments. But there is a large group of influential people - either coders or managers - who I assume would say I'm spouting lazy and reckless nonsense.

Here are some ways I have of putting it:

  • An extreme focus on unit tests - especially at the cost of pulling attention and resources from higher levels of testing - is like obsessively inspecting all your individual Lego Bricks for cracks and defects. Yes, it's important to be building with solid bricks, but honestly that's not where the problem in your structure is going to show up! 
  • Bugs are emergent properties! A well designed unit should be just about "too small to fail"
  • The problems I see actually show up in production are coders coding along the "happy path" - and then that same "happy path" is what is baked into the setup of the unit test. (I wish instead of testing the units, we had much more emphasis on firming up the contract between the unit and its environment...) 
  • And as for coders writing their own tests - it's very difficult to persuade someone to do a really focused search on something they don't really want to find - i.e. flaws in their previous work.
  • It's like testing 100 trees and saying "yup, I know this is a good forest"  
It's not a hill I'm willing to die on, I'm not going to be "That Guy" about it. If this is the Kool-Aid I'm being paid to drink, I'll have plenty of gulps. I understand there are some side benefits - it can get a coder to think more clearly about the inputs to the thing, it can make future refactoring feel safer, etc. But I'm going to be hard pressed to believe that that kind of benefit is worth the cost of making and maintaining this shadow implementation of every damn unit.  (And when I heard my manager's manager say "well we expect as unit coverage increases, the number of incidents in production will go down accordingly" I had to bite my tongue...)

I liked Mark Talbot's answer to the Quora "Why do many programmers think that unit testing is not worth it?" - in part -
The problem is that as soon as you replace some dependency with a stub or mock, you are no longer testing the system in the same way it works in production. All you have done is assume that your stub/mock behaves in the same way as the dependency it is replacing (at least as far as the unit under test is concerned) and then write your tests based on that assumption. But it is not valid assumption. If I come along and modify the real dependency (maybe it’s some 3rd party library that I’m upgrading) I have no idea at all whether your code really works with the new dependency, because all the calls to it in the unit tests are stubbed out. (And I’m not going to check all 500,000 lines of code manually - what is the point in automated testing if I need to do that?)
See also some bits I grabbed on Joel Spolsky on Complete Test Coverage...

Since some of the people who are unit test advocates are pretty sharp, I have to assume it's one of those "different ways of looking at the world" things - Douglas Hofstadter writes on holism vs reductionism and I think it may be one of those kind of things - reductionists assume it's mostly a matter of making sure each part works. But in general, I'm always more concerned about interactions - the surface of contact vs the essence of what something is in isolation... so I think unit tests are not worth the reputation they seem to carry.

No comments:

Post a Comment