Navigating Pitfalls in Quality-Driven Development Practices

In my previous post, we explored practices like observability, feature toggles, deployment automation, WIP limits, continuous deployment, TDD, pair programming, and trunk-based development - each essential for delivering quality at speed. However, implementing these practices can introduce new complexities if not managed carefully, ultimately undermining their effectiveness. Here, we’ll dive into common pitfalls with each practice and how to best avoid them.

Observability, when integrated thoughtfully, provides crucial real-time insights that keep applications stable. Yet without clear focus, it can quickly lead to information overload. With too many metrics and alerts, teams risk missing critical issues amidst the noise. To avoid this, observability efforts should concentrate on high-impact metrics directly tied to user experience, like latency, error/success rates, resource utilization, etc.¹ Here’s a simple example using Grafana:

Setting clear alert thresholds via SLO/SLI and regularly pruning outdated metrics ensures that observability remains a source of insight rather than distraction.

Alerts should be treated seriously: if an alert fires, it must signal a required action. Otherwise, you erode trust in the alerting system. It’s the classic “boy who cried wolf” scenario - too many meaningless alerts, and critical ones will be ignored².

Feature toggles offer flexibility in how and when a functionality is released, but they can quickly lead to clutter if not managed. Unused or stale toggles add complexity to the codebase, introducing unnecessary risk. To keep toggles efficient, it’s essential to have a strategy for toggle lifecycle management. Each toggle should be deleted the moment it’s no longer necessary. A good rule of thumb: if you’re considering building a system just to manage your feature toggles, chances are you already have too many - and aren’t cleaning them up when no longer needed.³

Feature toggles often go hand in hand with configuration toggles, which can introduce similar complexity if unmanaged. Before introducing a configuration please consider if it can be replaced by a sensible default. Example: when adding a configuration to decide if a set of users want to see a page with a list of orders, please take into consideration that it might be replaced by a simple check to see if the user has orders and show the page in this situation and hide if there are no orders to show.

Setting** Work-In-Progress (WIP) limits** can improve team focus and prevent multitasking, but overly strict limits can create bottlenecks, especially if they don’t reflect team capacity. Also, when defining WIP limits it’s important they are enforced and that team members work on the most important pieces of work prioritized by the team instead of just working on what they feel like or on the areas they are more comfortable with, the same with making sure that a task is complete before picking the next task on top of the TODO column. In addition, encouraging team members to support ongoing tasks before starting new ones fosters collaboration, allowing WIP limits to drive steady, efficient progress. Remember: to finish is more important than starting⁴. If these principles are not respected, the benefits of introducing a WIP limit are lost.

Pair programming fosters collaboration and knowledge sharing between two people; we can take these benefits one step further if we do frequent pair rotation (twice a week, for example) so that more people work on a given story when it takes several days to complete, and by sparkling the usage of mob/ensemble programming here and there for specific cases. The benefits can be extended even more if we encourage cross-team pairing sessions when there are problems worth solving that involve more than one team’s code base⁵.

When teams are starting with pair programming, a pair rotation sheet like the following might come in hand:

As the practice matures within the team, the rotation becomes more and more organic and this kind of sheet is no longer necessary.

Continuous deployment enables rapid, frequent releases, but without safeguards, it risks pushing unstable code to production. If we introduce continuous deployment without having confidence in the automated tests that are run in the CI/CD pipeline before deployment. It’s like jumping out of a plane without knowing if the parachute will open. The solution lies in pairing continuous deployment with robust testing, observability, and staging environments that mirror production. Using canary releases to test changes first with a subset of users can further reduce risks, allowing teams to deploy confidently without compromising quality⁶.

The focus on speed makes it tempting to try to have the teams with 100% capacity allocated and also to try to pre-assign specific tasks to certain people (example - John Doe is the expert on functionality A, so all improvements or bugs on this functionality are assigned to John Doe as he will get it done faster) two common anti-patterns that can have significant negative consequences:

• 100% capacity allocation not just incentivizes over-planning that gets in the way of quick feedback looks, but also prevents people from having the necessary breathing space essential for creativity to solve difficult problems and to come up with new ideas⁷.

• Pre-assigning tasks to specific people creates knowledge silos that can lead to teams depending on individual members which puts the teams (and the orgs) in very difficult situations if the people decide to leave, become sick, or go on holidays. To avoid this, we should foster collaboration and ownership on team, not individual ownership of certain tools or areas of the source code. Minimum level of delivery and ownership should be the team and not individual members of the team.

The same mindset often leads to team structures based on specialization rather than value streams. Having specialization teams leads to a considerable coordination overhead due to the number of dependencies between teams necessary to create user value - with dependencies come more communication lines that often lead to information being lost on the way. For product-driven organizations with a strong focus on delivering business value quickly teams formed around value stream teams have significant benefits⁸.

Finally it’s important that teams understand the importance of adaptation, not just on what they build, but also on how they work. All the practices described stem from strong first principles - but teams must regularly validate whether they still serve their context and be ready to adapt. All teams are different.

Footnotes

1. Burns, B., Beda, J., & Hightower, K. (2021). Kubernetes: Up & Running (3rd ed.). O’Reilly Media. ↩

2. Jones, A., & Kim, S. (2020). Effective Alerting Strategies in Distributed Systems. ACM Queue. https://queue.acm.org/detail.cfm?id=3454124 ↩

3. Fowler, M. (2010). FeatureToggle. martinfowler.com. https://martinfowler.com/articles/feature-toggles.html ↩

4. Reinertsen, D. G. (2009). The Principles of Product Development Flow: Second Generation Lean Product Development. Celeritas Publishing. ↩

5. Shore, J., & Warden, S. (2007). The Art of Agile Development. O’Reilly Media. ↩

6. Humble, J., & Farley, D. (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley. ↩

7. Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press. ↩

8. Team Topologies (Skelton & Pais, 2019). Organizing Business and Technology Teams for Fast Flow. IT Revolution Press. ↩