tech.reversedelay.net

Quality is relational

by DJ Ramones •

Avery Pennarun, the CEO of Tailscale, wrote a blog post a few months back that supposedly did the rounds online among tech circles. It begins, and is apparently titled, with the rule of thumb that “every layer of review makes you [ten times] slower”. Amazingly, his example of how this holds true from the first level of a 30-minute programming task all the way to 2.5-year timescales of corporate executive functioning resonates with my own professional experiences.

I suspect that many, if not most, of those who shared this post focused on the comments on AI, specifically on LLM-powered coding, about which the author does not hold a black-or-white position (though his “AI Developer’s Descent Into Madness” observation is hilarious); instead, he quite usefully identifies both a key benefit—that AI has made the production of code very cheap—and a key challenge—that the production of code itself is just one part of the pipeline of software delivery, as emphasized elsewhere by the prominent Django contributor James Bennett.

The more important points of the post, however, revolve around the question of quality assurance in software development, and deserve synthesis into two points.

One, that the culture of code reviews and quality assurance testing is broken. “Broken” is hyperbole, because code review obviously does provide value to the teams that do it. It does catch bugs before they are merged and deployed, and it does catch issues with requirements and specifications. The problem is the variability of its effectiveness. At worst, when the responsibility falls upon tired or demoralized team members, code review can become performative, just another process checkbox to be ticked with apathy.

Part of the fix is a change in mindset in what code reviews should aim for (perhaps just a reminder, as many software people would already know this, and might have simply forgotten under the pressures of the particular organizational cultures they have found themselves working in). Reviews should address root causes, or at the least, deeper causes, of problems, rather than particular instances of classes of problems:

The job of a code reviewer isn't to review code. It's to figure out how to obsolete their code review comment, that whole class of comment, in all future cases, until you don't need their reviews at all anymore.

(Think of the people who first created "go fmt" and how many stupid code review comments about whitespace are gone forever. Now that's engineering.)

The author also points out how quality assurance organization can be dysfunctional, in that the incentives can be misaligned and in fact counterproductive in terms of the ultimate goal of better-quality production, citing the case of two-pass QA organizations where the presence of additional checkers reduces the original producers’ desire to minimize defects right at the outset.

This leads to the second critical point: that a “real culture of quality” is founded on trust.

The basis of the Japanese [manufacturing] system that worked, and the missing part of the American system that didn’t, is trust. Trust among individuals that your boss Really Truly Actually wants to know about every defect, and wants you to stop the [production] line when you find one. Trust among managers that executives were serious about quality. Trust among executives that individuals, given a system that can work and has the right incentives, will produce quality work and spot their own defects, and push the stop button when they need to push it.

A genuine culture of total quality, in organizations, depends therefore on getting in place some deeply human and social factors. Factors such as psychological safety, where workers and team members are welcome and encouraged to make productive criticisms, and kindness or even mercy and forgiveness, where mistakes uncovered in the course of quality inspections are not held against the individual, where dips in key performance indicators are deliberately overlooked in the belief and hope of redemption.

The author wrote about trust, but implicit to that is a sense of ownership. It has been my experience that workers perform best when they exhibit a sense of ownership over whatever it is they are working on. I do not know if it is correlation or causation; if it is causation, I do not know which direction it flows: is it the sense of ownership that drives good performance, or does good performance spur a sense of ownership? In any case, I suppose it is not a stretch to claim that fostering good relations, a good social environment, would help uplift trust and responsibility and ownership, all these fuzzy factors that have been proven to improve production quality, be it on the factory floor or in software company offices.