Procedural debt: what is it and why care?

Excel Hell
The term ‘technical debt’ has been an established part of the software engineering industry for 20 years. It describes the series of short-cuts and trade-off’s that have been taken over time in order to get a software product ‘out the door’ quickly, but ultimately slow the project down as these short-cuts start to create problems and complications for the project over the longer term.

It is a statement of liability, just like the more traditional use of the word ‘debt’. It is the price you will have to pay in the future for the quick-wins you had today. And like financial debt, it comes with an interest bill. The cost of replacing a quick-fix or stop-gap measure with a quality solution increases as time goes by, as the detailed understanding of the problem fades from memory or people move to other job roles or organisations. It is the ticking time bomb that most business managers should be aware of.

While the software engineering industry has analysed this phenomenon extensively with a view to controlling it – comparatively little has been done in the business operations world, even though exactly the same thing happens. Instead of building up technical debt, your business operations build up ‘procedural debt’. And just like the other forms of debt, it can become a crushing limitation if not controlled.

What is procedural debt?

The concept of procedural debt is that there is a cost to taking short cuts (intentional debt) or making mistakes (unintentional debt) and that the cost of not dealing with these short cuts and mistakes will increase over time. No single item, or even small collection of items, inflicts a material cost. However, over time the number of items build up to be a significant cost, and crucially, the cost of fixing each item has increased as the knowledge required to fix it efficiently has faded or left the organisation.

Why should I care about it?

Like financial debt, procedural debt has a cost associated with it. With financial debt we know how much it would cost to pay off a debt today and we can calculate how much interest we will have to pay in the future. Procedural debt however is more difficult to quantify. We don’t really know how much debt we have taken on – you may have taken on a lot of unintentional procedural debt – and you may still be taking it on without knowing it. And we can’t quantify how much it is really costing us – how much interest we have paid so far, what the total cost may be in the future if we don’t take care of it today. There is a lot of uncertainty surrounding procedural debt:

cost of change

But we know it when we see it.

We find ourselves in environments where it is difficult to change established processes, but competitors seem able to change quickly. It is hard to find someone who really understands the details of how a process works. Or when key personnel are away, everything slows down or stops. There are errors in the data key processes produce, or, even if we cannot point to specific errors, we lack confidence in the outputs. We dread being asked to provide accurate documentation for the process, or to test if it is a genuinely repeatable process. Conversations that refer to compliance obligations from regulators are carried out in hushed tones or avoided. You get nervous when the auditors (internal or external) show up to examine how your part of the business operates. When business volumes grow, we find ourselves having to hire more and more people to perform all the manual work that has built up around the process.

Procedural debt is something we easily recognise if we care to look for it. The cost of (inevitable) change in the business just seems to be a lot more expensive and slower than it should be.

So where does procedural debt it come from?

Here is a list of the most common sources of procedural debt in our experience from working with clients who are working to automate their business processes. They are aware of procedural debt, and are taking steps to reduce or remove it from their business. This list is based on their experiences and reflections:

  • Failure to recognise an accumulation of gradual change – managers who have been in a business for a long period of time fail to notice the many small changes that have occurred and the process that is in place today is not only different from the one they ‘cut their teeth on’, but is now more complicated and more effort than it was in the past.
  • Quick fixes – in order to respond to a sudden change in business needs or practices, a quick fix will be implemented to a process. This might take the form of manually adjusting a spreadsheet at a certain point in the process, or manually excluding specific lines of data from a source system. In the following months, this “fix” is repeated, until it is considered “part of the process”. This happens in large organisations when the cost of getting a proper fix in place is high due to the need to engage IT departments and change application software.
  • VBA macros – are a great way to automate basic spreadsheet work, unfortunately they are also a great way to make what they do effectively a secret within your organisation. Unless your business staff are IT developers in their spare time, the chances are they do not fully understand all the things that are going on in the VBA macros they run. Furthermore, if your business staff are the people programming your VBA macros, you might want to examine if this IT development work is being reviewed and tested with the same discipline that is required of the IT department. One of the most common stories we hear is “the guy who wrote this macro has moved on, so we cannot change our spreadsheets because we don’t know if it will break, and we don’t know how to fix it if it does”.
  • Documentation is inaccurate or missing – No-one ( in a healthy state of mind ) enjoys documentation. However it is still amazing how many organisations knowingly run their business without it for key processes. This happens because people make changes to a business process and (a) do not recognise that they should be documenting it, or (b) they are too rushed and short of time to document it, or (c) they just don’t care enough to document it, and they know there are no consequences for ignoring the documentation. Particularly in a post Sarbanes-Oxley world (remember that?) where many organisations are legally required to keep accurate documentation of key processes, it still doesn’t happen.
  • Mergers and acquisitions – are great sources of procedural debt. Everything changes overnight. Product lines are merged, reporting consolidated, people entering and leaving the organisation. When the dust settles, people look around and discover that at best they only understand a subset of the new XYZ process – because it has now changed to accommodate the “new” business, and they don’t really understand how that business works. Can they simply refer to the documentation for the processes in question?, nope, see the point above.
  • New and changed products – this is similar to mergers and acquisitions, in that the nature of the data flow through business is changing when products are added or removed from a line-up. What makes this an interesting dynamic is that for many businesses this is becoming a more frequent event, and a more complex event, as the sophistication of products is increasing. In the insurance industry, some clients we work with accommodate a new insurance product every month. That requires changes in most of the key reporting processes because they are working with different product codes and different sources of data. The business processes that are used to aggregate and report management information are struggling just to keep up – let alone get on the front foot and add insight to the raw data.
  • No mechanism for testing and verifying processes – it is foolish to assume that something with any complexity will work perfectly the first time. However many businesses operate with this assumption when they make changes to their processes. They change the way data is collected, manipulated and distributed, yet only perform a cursory amount of verification work to prove that the modified process is correct. Part of the reason for this is that manually intensive processes cannot simply be performed multiple times at will because they take 2-3 days to run each time. It would be cost prohibitive, let alone drive people mad. Another factor is the mindset that is being applied to the business process. Some managers just don’t recognise that a process, whether it is software or human driven, can be wrong, and needs to be verified.
  • Dependence on manual procedures – maybe one of the most significant factors is the reliance on people to perform repetitive data manipulation tasks. This is how human errors creep in, including the famous “cut and paste” error. People get bored doing dull spreadsheet work, and when people get bored, they make mistakes. Even the best people. Furthermore, there are natural limits to what a human can do. It is not uncommon for our clients to be working with spreadsheets that have 200,000 rows and 100 columns. There is no practical way for a human to scan this range of numbers and quickly look for formula errors or values that are significantly outside of a “rule of thumb” range, however the business process in use still has people doing this sort of “sanity check” review.

There are probably several more sources of procedural debt. If you have encountered any, please drop me a note or comment below. I am interested in your experiences.

So what can we do about it?

In the next article on this topic we will be looking at what business managers can do about procedural debt. There is no magic answer, but there are some basic principles that can be applied that help an organisation to substantially reduce and minimise their procedural debt.

6 Comments
  1. Another common scenario is that in many organizations, client base grow gradually thus work load but number of staff members remain unchanged. Consequently overhead work load & pressure to achieve team goal increases and amplify the probabilities of human errors. This situation leads the team to introduce additional spreadsheets and use of different software to minimize the chance of errors and reduce the pressure for time being. This scenario repeats time to time and situation goes out of control and introduces numerous high level problems such as growing numbers of spreadsheets, key person dependency, audit-ability, lack of process documentation, no backup plan and seniors lose control over knowledge of actual business process. Senior management often hesitate to involve in the processing staff, consequently organization gains more procedural debt.

    From my point of view, procedural debt affects the talented employees of organizations such as:
    -> Talented employees become reluctant to express their opinion for improvement,
    -> Lose the ability to lead the change,
    -> Increase the gap in common understanding of process between employees,
    -> Lose the focus from core objective of the role and get busy into processing & managing the data.
    -> Kills efficiency, intellectual thinking power, creativity and innovation.

  2. I think that this is a very insightful analysis of, in my experience, one of the most overlooked and neglected “overheads” in growing/changing businesses. I have seen it occur many times. I agree with all of the points above and in particular, the point about documentation. It is probably the hardest component to acheive, yet it is probably the most important “safety” mechanism a company can implement to reduce procedural debt. The other component that I think is important is training and sharing of knowledge between personnel within a company so that the knowledge base is widened, therefore reducing the procedural debt when some staff members move on.

    • The point on training, and the need to support staff transition and handover is a good one Sue. This article overlooked that. I might put that topic aside as the focus for an entirely separate post.

  3. As mentioned in the article, procedural debt, much like financial debt, is a liability. Businesses that take on financial debt borrow cash, and are liable to pay back cash. Businesses that take on procedural debt borrow time, and are liable to pay back time. ‘Interest payments’ become due whenever an employee becomes stuck in trying to understand the roles of all the short-cuts and trade-offs. Ironically, the ‘principal’ is added to whenever an employee does not realise that these short-cuts and trade-offs exist; therefore, the ‘principal’ is being added to continuously. The ‘principal’ is the time required to reverse the accumulated mass of problems caused by the short-cuts and trade-offs (might I also remind readers of the role of short-cuts in JP Morgan’s $6bn “London Whale” loss). However, the purpose of operating a business is not to generate time. The purpose is to generate cash. No matter how successful a business becomes, it will never be able to generate time. There is no product that a business can sell, nor service that a business can provide, in return for time. The consequences of taking on more and more procedural debt are therefore dire. To avoid sounding too apocalyptic, I must stress that is it possible for businesses to deploy long-term, sustainable, solutions to alleviate their procedural debt. Unfortunately, given the current range of technologies available to mankind, time machines are not the answer. The answer lies in process enhancement and automation.

  4. Great post. It’s almost like death by a thousand cuts…

    For me, the turnover of staff coupled with a lack of consistent high-quality documentation compounds the procedural debt.

    For example, Person A starts their new job at Company X:

    -> They inherit an Excel workbook with formulas and macros used to produce a report
    -> There is no documentation and their predecessor who was responsible for the process has already left the company
    -> They receive a less-than-perfect description of the process from someone who has a less-than-perfect understanding of the process
    -> They are told to not only produce the report every month, but they need to change the format

    What does Person A do?

    -> Most likely, take the process they don’t fully understand and tweak it in whatever way they can in order to get the output format they need.
    -> They add new sheets and calculations to the Excel file, ignoring old ones – but not deleting them because they are afraid that doing so will ‘break something’
    -> They constantly fix errors… in some cases the fix becomes part of the process itself, e.g. override this number because it’s wrong

    … Person A leaves the company. Person B is hired and takes over the process and the cycle repeats itself, each time adding to the debt. Now multiply this across the hundreds if not thousands of processes across the organisation.

    Documentation in theory is great, however, asking staff to keep these up to date and of a certain level of quality is difficult in practice. If I am the only person that knows the specific details for a process, how can I be made accountable for “up-to-date” documentation when I am the only one that knows what “up-to-date” is. Furthermore, the quality of documentation will vary greatly from person to person, just like all things in life, some people will be really good at it and others will be “less-than-perfect”…

  5. My understanding of procedural debt is similar to getting a loan from a bank. When you borrow a loan from bank, you can used it to invest in other high margin area to get more money. However, you need to make sure pay off the repayment on time to minimize the interests.

Leave a Reply

Latest Tweets

The age of #Robots is upon us! Everything you need to know about #RoboticProcessAutomation #RPA -… twitter.com/i/web/status/8…

reply · retweet · favorite

Newsletter

By signing up, you agree to our Terms of Service and Privacy Policy.

Address

Suite 1506, Westfield Tower 2, 101 Grafton St Bondi Junction NSW 2022 Australia

View Larger Map

Contact Us

AUS: +61-2-9386-0202
UK: +44-20-8123-1251
Send us an email

ContactUs.com
WordPress Image Lightbox