Procedural Debt: What is it and Why Should you Care?

The term 'technical debt' has been an established part of the software engineering industry for 20 years. It describes the series of short-cuts and trade-off's that have been taken over time in order to get a software product 'out the door' quickly, but ultimately slow the project down as these short-cuts start to create problems and complications for the project over the longer term.

It is a statement of liability, just like the more traditional use of the word 'debt'. It is the price you will have to pay in the future for the quick-wins you had today. And like financial debt, it comes with an interest bill. The cost of replacing a quick-fix or stop-gap measure with a quality solution increases as time goes by, as the detailed understanding of the problem fades from memory or people move to other job roles or organisations. It is the ticking time bomb that most business managers should be aware of.

While the software engineering industry has analysed this phenomenon extensively with a view to controlling it - comparatively little has been done in the business operations world, even though exactly the same thing happens. Instead of building up technical debt, your business operations build up 'procedural debt'. And just like the other forms of debt, it can become a crushing limitation if not controlled.

What is procedural debt?

The concept of procedural debt is that there is a cost to taking short cuts (intentional debt) or making mistakes (unintentional debt) and that the cost of not dealing with these short cuts and mistakes will increase over time. No single item, or even small collection of items, inflicts a material cost. However, over time the number of items build up to be a significant cost, and crucially, the cost of fixing each item has increased as the knowledge required to fix it efficiently has faded or left the organisation.

Why should I care about it?

Like financial debt, procedural debt has a cost associated with it. With financial debt we know how much it would cost to pay off a debt today and we can calculate how much interest we will have to pay in the future. Procedural debt however is more difficult to quantify. We don’t really know how much debt we have taken on – you may have taken on a lot of unintentional procedural debt – and you may still be taking it on without knowing it. And we can’t quantify how much it is really costing us – how much interest we have paid so far, what the total cost may be in the future if we don’t take care of it today. There is a lot of uncertainty surrounding procedural debt:

But we know it when we see it.

We find ourselves in environments where it is difficult to change established processes, but competitors seem able to change quickly. It is hard to find someone who really understands the details of how a process works. Or when key personnel are away, everything slows down or stops. There are errors in the data key processes produce, or, even if we cannot point to specific errors, we lack confidence in the outputs. We dread being asked to provide accurate documentation for the process, or to test if it is a genuinely repeatable process. Conversations that refer to compliance obligations from regulators are carried out in hushed tones or avoided. You get nervous when the auditors (internal or external) show up to examine how your part of the business operates. When business volumes grow, we find ourselves having to hire more and more people to perform all the manual work that has built up around the process.

Procedural debt is something we easily recognise if we care to look for it. The cost of (inevitable) change in the business just seems to be a lot more expensive and slower than it should be.

So where does procedural debt it come from?

Here is a list of the most common sources of procedural debt in our experience from working with clients who are working to automate their business processes. They are aware of procedural debt, and are taking steps to reduce or remove it from their business. This list is based on their experiences and reflections:

• Failure to recognise an accumulation of gradual change - managers who have been in a business for a long period of time fail to notice the many small changes that have occurred and the process that is in place today is not only different from the one they 'cut their teeth on', but is now more complicated and more effort than it was in the past.
• Quick fixes - in order to respond to a sudden change in business needs or practices, a quick fix will be implemented to a process. This might take the form of manually adjusting a spreadsheet at a certain point in the process, or manually excluding specific lines of data from a source system. In the following months, this "fix" is repeated, until it is considered "part of the process". This happens in large organisations when the cost of getting a proper fix in place is high due to the need to engage IT departments and change application software.
• VBA macros - are a great way to automate basic spreadsheet work, unfortunately they are also a great way to make what they do effectively a secret within your organisation. Unless your business staff are IT developers in their spare time, the chances are they do not fully understand all the things that are going on in the VBA macros they run. Furthermore, if your business staff are the people programming your VBA macros, you might want to examine if this IT development work is being reviewed and tested with the same discipline that is required of the IT department. One of the most common stories we hear is "the guy who wrote this macro has moved on, so we cannot change our spreadsheets because we don't know if it will break, and we don't know how to fix it if it does".
• Documentation is inaccurate or missing - No-one ( in a healthy state of mind ) enjoys documentation. However it is still amazing how many organisations knowingly run their business without it for key processes. This happens because people make changes to a business process and (a) do not recognise that they should be documenting it, or (b) they are too rushed and short of time to document it, or (c) they just don't care enough to document it, and they know there are no consequences for ignoring the documentation. Particularly in a post Sarbanes-Oxley world (remember that?) where many organisations are legally required to keep accurate documentation of key processes, it still doesn't happen.
• Mergers and acquisitions - are great sources of procedural debt. Everything changes overnight. Product lines are merged, reporting consolidated, people entering and leaving the organisation. When the dust settles, people look around and discover that at best they only understand a subset of the new XYZ process - because it has now changed to accommodate the "new" business, and they don't really understand how that business works. Can they simply refer to the documentation for the processes in question?, nope, see the point above.
• New and changed products - this is similar to mergers and acquisitions, in that the nature of the data flow through business is changing when products are added or removed from a line-up. What makes this an interesting dynamic is that for many businesses this is becoming a more frequent event, and a more complex event, as the sophistication of products is increasing. In the insurance industry, some clients we work with accommodate a new insurance product every month. That requires changes in most of the key reporting processes because they are working with different product codes and different sources of data. The business processes that are used to aggregate and report management information are struggling just to keep up - let alone get on the front foot and add insight to the raw data.
• No mechanism for testing and verifying processes - it is foolish to assume that something with any complexity will work perfectly the first time. However many businesses operate with this assumption when they make changes to their processes. They change the way data is collected, manipulated and distributed, yet only perform a cursory amount of verification work to prove that the modified process is correct. Part of the reason for this is that manually intensive processes cannot simply be performed multiple times at will because they take 2-3 days to run each time. It would be cost prohibitive, let alone drive people mad. Another factor is the mindset that is being applied to the business process. Some managers just don't recognise that a process, whether it is software or human driven, can be wrong, and needs to be verified.
• Dependence on manual procedures - maybe one of the most significant factors is the reliance on people to perform repetitive data manipulation tasks. This is how human errors creep in, including the famous "cut and paste" error. People get bored doing dull spreadsheet work, and when people get bored, they make mistakes. Even the best people. Furthermore, there are natural limits to what a human can do. It is not uncommon for our clients to be working with spreadsheets that have 200,000 rows and 100 columns. There is no practical way for a human to scan this range of numbers and quickly look for formula errors or values that are significantly outside of a "rule of thumb" range, however the business process in use still has people doing this sort of "sanity check" review.

There are probably several more sources of procedural debt. If you have encountered any, please drop me a note or comment below. I am interested in your experiences.

So what can we do about it?

In the next article on this topic we will be looking at what business managers can do about procedural debt. There is no magic answer, but there are some basic principles that can be applied that help an organisation to substantially reduce and minimise their procedural debt.