Writing · AI and technology leadership

Where technical debt becomes expensive

Technical debt costs most in hardware, in architecture and at the boundaries between teams.

Technical debt is not one thing. Working across hardware, firmware and software teaches a leader that it is several different things, with very different repayment costs depending on where in the stack it has accumulated.

In software, debt tends to be expensive but recoverable. A poorly structured codebase can be refactored. A bad architecture decision can be unwound, usually at the cost of engineering time and some disruption to delivery. The debt is real, but the tools for addressing it exist, and repayment, while painful, is usually survivable.

In hardware, the dynamics are completely different. A circuit board designed around a constraint that turns out to be wrong does not get patched. It gets respun, which means tooling costs, lead times, component qualification and, in the worst cases, a field recall. The cost of debt at the hardware layer is not engineering time. It is money, market delay and customer trust.

I have seen technically sound firmware decisions rendered useless by a hardware choice made six months earlier that closed off the option the firmware team needed. The debt was invisible when it was taken on, and very visible once it came due.


Where the expensive debt accumulates

In multi-discipline systems, the most dangerous debt is always at the boundaries: where hardware assumptions meet firmware requirements, where firmware timing meets software expectations, where the local system architecture meets the cloud. Debt that sits cleanly inside one layer can often be managed by that team. Debt at the boundary between layers tends to be discovered later, by the wrong people, at the worst moment.

Architecture decisions create the longest-lasting debt of all, because everything else is built on them. The communication protocol, the partitioning of intelligence between edge and cloud, the security model, the update mechanism: these decisions are made early, often under pressure, with incomplete information. Once they are embedded in a shipped product, changing them is not a refactor. It is a redesign. In connected products, where a security decision made at architecture stage may need to hold for ten years of field operation, the cost of getting it wrong is not academic.


Deliberate debt and accidental debt

The distinction that matters most in practice is not between debt and no debt. It is between debt that is known and debt that is not. Every product ships with some technical debt. The question is whether the team knows where it is, how much it will cost to service and when it will come due.

Deliberate debt is a legitimate tool. Taking a shortcut at the firmware level to hit a launch date, with a clear plan to address it in the next sprint, is a calculated trade-off. It is manageable because it is visible.

Accidental debt is a different problem. It accumulates because nobody thought carefully about the architecture, because a cross-discipline constraint was missed, or because speed felt more important than rigour at a moment when rigour was actually critical. It compounds silently and tends to surface at the worst possible time.

At its heart, the technology strategy conversation is about keeping enough visibility to control the first kind of debt and to stop the second kind accumulating in the first place.


The deficit that comes from not innovating

A third kind of technical deficit gets less attention than it deserves. It accumulates not from moving too fast but from refusing to move at all. Companies that habitually copy rather than lead, that wait for competitors to prove a technology before adopting it, and that treat conservatism as a form of risk management end up with their own version of the problem. They are always eighteen months behind the companies that did the hard work first. Their architecture reflects someone else's decisions, made in a different context for a different product. And they never develop the internal capability that only comes from having built something genuinely new.

This is a strategic choice with real costs. The risk of moving early is that the company sometimes backs the wrong technology and pays to change course. The risk of always following is that the company never leads, its competitive position is permanently reactive, and the engineers who want to build new things leave for companies that let them. Neither extreme is viable. The judgement lies in knowing which bets are worth taking early and which are better left until the market has done the proving.


The balance in practice

Across the products I have built and the companies I have worked in, the approach that has served me best is to be deliberately strict about architecture decisions and deliberately flexible about implementation decisions. The architecture sets the constraints within which implementation debt can be managed. If the architecture is sound, local shortcuts can be tolerated and addressed. If it is wrong, no amount of clean implementation will fix the problem.

That means investing the time to get the cross-discipline architecture right before committing to implementation, even when the pressure to move quickly is strong. Undoing an architecture decision in a shipped product is almost always more expensive than the time it would have taken to get it right at the outset. It also means keeping an honest view of where the current debt sits, at which layers, and what its repayment schedule looks like, so that the strategy can accommodate it rather than be surprised by it.

Technical debt in a multi-discipline product will never be zero. The goal is to know where it is.


Four things worth taking seriously

For boards: ask where the product's technical debt sits and at which layer. Debt in hardware and architecture is repaid in money, market delay and customer trust, not engineering time.

For engineering leaders: watch the boundaries between hardware, firmware, software and cloud. That is where debt is taken on without anyone owning it, and where it is found last.

For founders: be strict about architecture and flexible about implementation. A shortcut in the code can be repaid. A wrong architecture in a shipped product means a redesign.

For anyone setting technology strategy: always following is also a form of debt. Decide deliberately which bets to take early, rather than waiting for competitors to prove every one.


Where in your own product does the debt sit that nobody has written down, and which boundary between teams is it hiding on?

© 2024 Catherine Ives-Yim. All rights reserved.

Catherine Ives-Yim

Catherine Ives-Yim

Chartered Engineer and independent technical adviser, with a lifetime at the bleeding edge of embedded systems, connected products, data platforms and AI-assisted engineering, who has advised clients across the UK, Europe, the Middle East, the Far East, North America and Africa. Based in Leeds.