Technical Debt, a term coined by Ward Cunningham, was originally defined in the early 90s to explain how, in extreme programming, you start with “writing code to reflect your current understanding of a problem even if that understanding is partial” which, according to his debt metaphor, allows you to take on debt “by developing software that you don’t completely understand, you are wise to make that software reflect your understanding as best as you can, so that when it does come time to refactor, it’s clear what you were thinking when you wrote it, making it easier to refactor it into what your current thinking is now.”, thus paying down that debt. However, it has come to mean a lot more than that in the years since, as the concept applies to design, documentation, infrastructure, and security decisions as well.
A manageable technical or code debt is a good thing as it speeds development. There are also instances when the speed to market for a critical feature, bug fix, or other short-term priorities trump the analysis of medium to long-term impact. The key word here is “manageable”. If left unchecked, it becomes a problem of exponentially increasing magnitude with time when new debt is accumulated on top of existing one. Not to mention the human cost of technical debt.
If a team is always under pressure to deliver under unrealistic timelines then a review of the overall business practices should be in order but that is not always possible due to constant demands of keeping the users, clients, and shareholders happy (or at least placated enough to not jump ship).
Then there are situations where, due to a lack of resources, the team is compelled to take shortcuts and cut corners in implementing a solution that may not be optimal in the long run.
Not to mention that one manager, who assumes that the only way to boost productivity is to artificially create challenges with unrealistic deadlines and constant pressures. But that is a topic for another time.
In any case, with or without the justifiable urgency or artificially manifested pressures, the lure of “quick & dirty” is often too strong. However, quick and dirty solutions are not the only cause for accumulating technical debt. Some of the other causes are:
- Lack of experience or understanding: Developers who are unfamiliar with a programming language, technology, or best practices may make decisions that generate code debt. Similarly, a lack of understanding of the requirements can lead to suboptimal implementations.
- Changing requirements: Changes in requirements or scope can result in teams needing to make quick changes or add features that may create a maintenance nightmare in the future.
- Constantly evolving technology and frameworks: As technology and frameworks evolve, older code may become outdated and require updates or changes to maintain compatibility or take advantage of new features. If this is not done in a timely manner, it leads to more debt. It is often the approach to just deal with breaking changes.
- Legacy systems: Working with older, poorly documented code makes it difficult to understand and update, which results in new code that generates debt.
- Neglecting code quality: Unfortunately, I have come across this more often than I would like to believe and it always leads to issues later on.
Prudent technical debt is a necessary and integral part of the system design, development, and deployment. However, if not addressed in time, it will result in a constant increase in the cost of maintaining or updating the codebase as more changes are made, leading to systems that become more complex and difficult to work with over time.
Given that it is such an integral part of software engineering, systems design, and maintenance, the real issue is not technical debt itself but the lack of planning for it and sometimes actively ignoring it. Globally, developers spend about 30% of their time on maintenance related to technical debt and “bad code” costs over $85 billion annually (2018 estimate).
The question then arises, how to best manage it?
The very first step for managers as well as engineers, is to educate themselves about the risks associated with design and implementation choices from a technical debt perspective.
Managing technical debt is an ongoing process and requires commitment, discipline, and collaboration between teams. Continuous and iterative processes have to be implemented to pay it down gradually before it becomes unmanageable. This is an essential part of a system with a healthy and maintainable codebase.
Some of the ways that you can achieve this are:
- Regular code reviews: Frequent code reviews, including discussions about code quality, coding standards, adherence to best practices, and design decisions, should be part of the process and must be budgeted as such. Similarly, code refactoring efforts should be prioritized to not only improve performance but also to improve readability and maintainability. If it is not planned and budgeted then it will not get done.
- Documentation: Maintaining inline documentation, API documentation, and updating architectural diagrams are some of the things that will require the least amount of effort when bundled with the change itself. The longer it is left the higher the cost will be.
- Linters and automated tests: The use of static code analysis tools, linters, and comprehensive automated tests help identify regression issues and code smells. Integrating Continuous Integration (CI) and Continuous Deployment (CD) pipelines in your workflow to automate this will help catch issues early.
- Plan for it: Create and maintain a backlog of technical debt items. These items must be prioritized along with new feature development. Technical debt starts to become an issue when it is ignored or addressed as an afterthought. The Debt Quadrants method, as defined by Martin Fowler, is an excellent tool to classify these items.
- Culture: Foster a culture of learning and improvement. This requires investment in training and skills development to keep the team up-to-date with evolving methodologies, modern technologies, and best practices.
- Avoid shortcuts: This is not always possible, but resist the temptation of quick and dirty solutions unless it is necessary. Shortcuts should be a deliberate decision instead of the norm. When you do have to take shortcuts, document it and add it to the technical debt backlog so that it gets addressed as soon as the pressure is off.
- Educate and Collaborate: Fostering an environment of openness and collaboration within the engineering organization goes a long way and is one of the key aspects for identifying issues early on. Outside of the core engineering teams, the magnitude of technical debt is not understood well oftentimes. All the stakeholders must be kept aware of the significance of the initiatives to pay the debt down to healthy levels. Collaboration between teams is the key.
There are also times when the technical debt has been allowed to accumulate for way too long or you are inheriting a system, through a new job perhaps, that has massive technical debts. This can be challenging but with a systematic approach, significant improvements can be made over time:
- Understand and document the extent of the problem: A thorough analysis of the codebase, architecture, design patterns, critical components, and stakeholder feedback will help identify areas of technical debt like outdated libraries, deprecated API usage, poorly structured code, outdated or missing documentation, etc. Historic context around design decisions should also be noted to get a deeper understanding.
- Classify: In-depth analysis will help you classify the issues and create Debt Quadrants. This is a very effective tool for prioritizing and decision-making.
- Prioritize: Identify areas that require immediate attention and those that can be addressed over time. Break these down into smaller more manageable tasks so that you can plan and prioritize based on the size, need, critical dependencies, impact, and alignment with other new feature development.
- Implement the best practices listed earlier to minimize regression issues. This may require buy-in from the engineering teams as well as other key stakeholders.
A thorough analysis, prioritization, and planning will lead you to a decision and tradeoffs. A complete rewrite may turn out to be the only logical solution in extreme cases. It seems daunting but could be an easier and more cost-effective option so keep an open mind and let the process guide you.
Effective technical debt management contributes to the long-term success, scalability, maintainability, and sustainability of software projects. It cannot be an afterthought and must be budgeted for in the early stages of the project so that the implementation decisions and tradeoffs can be tracked and addressed before it is too late.
Technical debt arises from shortcuts and compromises in software development. If left unchecked, it can hinder progress and lead to maintenance challenges and drain development resources. To manage it, acknowledge its existence, prioritize tasks, maintain good practices, document changes, and foster a culture of openness and improvement. Inherited code debt requires a systematic approach involving understanding, prioritization, and possibly even a complete rewrite to address it effectively.
Author: Ajay Maru, Senior Advisor
