Death to Excel! Long Live Excel!

[custom_frame_left] Excel Hell
[/custom_frame_left] The widespread use of Excel in every part of business has created a phenomenon called ‘Excel Hell’. This is the confusion and frustration of not knowing which file is the right version to use, not being confident that it is secure and not having a sense of control over how it is used. In this article we look at why ‘Excel Hell’ happens, what the root causes are – and what can be done about it.

Introduction

Automating business processes in 95% of organisations involves working with data in an Excel format. When you talk to enough businesses about automating processes, you inevitably find yourself having the famous ‘Excel Hell’ conversation. This involves hearing about the pain, frustration, fear, despair, anger and confusion of the IT department and/or technology director when it comes to managing the organisation’s Excel data.

Admittedly, having worked for Microsoft for 12 years, I was exposed to more than the usual share of these conversations. It is usually the IT Director or head of technology who speaks with the most venom about Excel. I find this puzzling because, if you had to list the most potent enabling desktop software technologies, surely Excel would feature in that list – and technology groups admire technologies that enable people right?

Well, in this case – no, they don’t. Here is a small sample of the quotes that capture the emotion surrounding this topic:

Excel is terrible for IT standards because staff keep creating their own spreadsheets to analyse and report with

We probably have more work taking place in Excel than in all of our sanctioned corporate IT systems combined

We have an IT strategy to remove Excel from the organisation – we will be spreadsheet free in 2 years

Excel is everywhere – every part of our business runs on Excel in some way or another – but we can’t keep track of it

and my favourite because of how much it unintentionally reveals:

We don’t like Excel because it moves too much decision making from the IT department to the end user

I find it fascinating that one of the most useful, widespread and impactful software technologies of the last 25 years generates this sort of commentary. It is also when I consider that working constructively with Excel is essential when automating business processes. SolveXia is in the business of providing organisations with a way to automate processes – so I thought it worthwhile digging deeper to discover why Excel Hell happens, and what can be done about it?

Why does Excel Hell happen ?

If it causes so much pain, why is Excel so ubiquitous? It cannot be just because it is installed on most corporate desktops. Availability alone does not auto-convert into usage. There must be reasons why people pick up Excel and do so much with it. I asked quite a few people, and this is what I heard:

  • I can do things in Excel that our corporate systems can’t do
  • I need my information structure in a certain way – and Excel is the fastest/cheapest way for me to get it like that
  • I need my data from different corporate systems integrated – and I can get this done in Excel
  • I can do ‘what-if’ analysis using Excel quickly and cheaply
  • I can get some of my IT needs met without having to engage the IT department if I do it in Excel

In summary, it seems people use Excel to complement the systems and services that the IT department ordinarily provides. They can do things in a ‘self-service’ fashion which is faster and easier for them, and people like autonomy. This explains why people use it, and the fact that so many roles within an organisation nowadays are working with information in various states of readiness explains why so many people use it.

So what are people really saying ?

As with solving most problems, you need to look and listen for the real issues that are driving the human (and often emotional) response. That is, look beyond the symptoms, and search for the root cause. These expressions of frustration stem from more substantive, and very real, problems. It should be possible to build a “phrase translator” that summarises what people say and what people mean … and this is what I have so far.

What is said What is meant
There are too many spreadsheets in our organisation and we never know which is the ‘source of truth’ We do not have a way to track and version our important files
People keep critically important Excel spreadsheets on the C drive and the business never knows where this data really is We do not have an effective approach to centrally storing and sharing our most important spreadsheets
People keep creating new spreadsheets and we have no control over the quality and correctness of the mini-applications that they build in Excel We do not have an effective approach to controlling the quality of the applications and processes that run our business
People are doing work in Excel that should be done in our core systems Our core systems are harder to work with than Excel, so people avoid them, preferring to work with what they feel makes them more productive
People use low quality cobbled together spreadsheet applications and processes instead of engaging IT and getting an enterprise quality solution IT is either too expensive, to slow, or too hard for our staff to engage as means of fulfilling their business needs
We have sensitive data sitting in Excel that is not secured or encrypted as it just sits on people’s PC’s in directories and emails – this poses a serious security risk We do not have the systems in place to secure our sensitive data
We have sensitive data Excel that is being shared between team members using USB keys and email – which can be lost or sent to the wrong person accidently – this poses a serious security risk We do not have the systems in place to share our sensitive data between staff members securely

Is there a way to deal with Excel Hell?

Yes. We have seen a number of our clients adopt strategies and tactics that manage these issues. Furthermore, we make addressing these issues an integral part of how we design our service – so that we can actively help our clients avoid or escape Excel Hell.

  • Adopt a ‘manage it’ instead of a ‘fight it’ mindset. Some organisations, and some people, are inclined to fight the widespread use of Excel. This seems short sighted because it discounts the significant benefits Excel can offer. An alternative mindset is to seek to manage spreadsheets more actively, trying to put in place systems that retain the good things about Excel, while simultaneously reducing the impact of the bad. This is really a difference in attitude and approach more than a particular solution. When you speak to business and IT executives, you can tell pretty quickly if they are in the ‘fight it’ or ‘manage it’ mindset. The later tends to be more constructive.
  • Think of Excel as three different products – and seek solution strategies for each. One way of thinking about Excel is that it is really 3 different applications/solution in one: It is used for (a) data storage (b) performing calculations and analysis, and (c) reporting and presenting information.Dissecting your organisation’s use of Excel along these lines can be helpful when you try to find ways to manage and mitigate the problems Excel Hell introduces. For example, by identifying those spreadsheets that are acting as data stores, you automatically highlight the set that most needs to be integrated and reconciled with established corporate data stores, and brought into version control. By identifying those spreadsheets that are primarily performing calculations and analysis, you have identified candidates for integration or inclusion in the corporate approach to application services. Identifying the spreadsheets that are primarily presentation and reporting, you have uncovered additional reporting requirements/components for the corporate reporting strategy.Inevitably organisations have spreadsheets that perform 2 or all 3 of these functions. This warrants furthers analysis as Excel in this case has become a broader delivery platform – and it is wise for the IT department to understand the business needs that are driving this.
  • Establish a central repository that has access and audit control for critical spreadsheets. One of the great advantages of Excel is that being a single file format, it can be versioned easily. There are plenty of inexpensive file versioning control systems in the market that are easy to setup and use. Beyond simple file versioning, there are ways to integrate Excel into a versioned and auditable process – while it continues to participate in business processes.
  • Address the real substance of the frustrations. Most of our clients express some level of frustration with Excel Hell, and what I tell them is that instead of trying to ban or get rid of Excel, it is usually possible (and easier) to address the root cause of the frustration. Without meaning to make this article sounds like an advertisement, I will explain how we use the SolveXia platform we sell on a SaaS basis to complement existing corporate IT systems, and address the primary issues:
    Problem How the SolveXia platform addresses this
    We do not have a way to track and version our important files The key here is to provide a single and simple system for storing Excel files that provides versioning. If this system is simple enough to use, and provides additional value around the Excel file, people will use it. We show clients that our platform is a simple, friendly and web based system that allows people to store their Excel files with versioning. They can always find the latest version of file, and be confident that it is the latest version. They can go back to prior versions if they need to – confident that the timeline of versioned files is accurate and reliable.
    We do not have an effective approach to centrally storing and sharing our most important spreadsheets We have chosen to address this issue in a very similar way to the issue above – provide a single simple system that people can store their important files in. The critical difference is that this is not like a LAN drive or file share (this approach will become a dumping ground). In the SolveXia platform everything is stored and organised in the context of a “business process”. You cannot just dump files randomly – they are attached to a modelled business process such as ‘produce end of week debtors balances’ or ‘reconcile invoices to the GL’. By requiring the storage of files in this way, we have found that sharing and communicating becomes more effective because it occurs at the business process level – which will nearly always include many files – rather than piecemeal file by file.
    We do not have an effective approach to controlling the quality of the applications and processes that run our business There are (at least) two aspects to this. When those responsible for process governance ask for control, they are asking for (1) complete information combined with (2) contextual information. Both of these need to be addressed in full if you going to deliver a sense of control and confidence.

    In SolveXia we approached this request by building in what we call ‘automated documentation’ – the ability for our clients to click on a process and generate a single document that describes every step and every data item in the process – expressed in ‘plain English’. Technology has advanced sufficiently that this can be generated on demand, so it is guaranteed to be accurate. There is no risk of human error (writing down the wrong thing, or not writing down everything). As a result everyone see and review the logic that is contained within a process, and all the data sources it uses, and how it uses them. In this way the application (business process) can be reviewed with complete and contextual information.

    Our core systems are harder to work with than Excel, so people avoid them, preferring to work with what they feel makes them more productive Fundamentally the only way to address this issue is to increase the priority and presence of self-service flexible applications. Most corporate IT systems have a very specific purpose (ERP, CRM etc…). They work well provided the business process stays within the understanding of the corporate system of how that process should be run. Inevitably however, that nature of business forces changes in process that test the limits of these large system. They were designed to be great specialists – not great generalists.We have tried to address this issue by providing clients with a system that is “general” in nature – that they can construct, modify and experiment with business processes with the same ease and speed that they can modify Excel spreadsheets. The key here is ensure that this general system has all the audit, transparency and security built into it. We believe that with the continuous pressure in business to do more with less, grow revenues and constrain costs – people will always look to experiment and innovate with the applications that support their business – you won’t be able to stop (and you probably don’t really want to). Rather than resist this, we focus on providing a platform that continues to deliver the flexibility of Excel but with the control attributes more typically associated with ERP, CR
    IT is either too expensive, to slow, or too hard for our staff to engage as means of fulfilling their business needs. Again there are (at least) two issues here. The first is cost and the second is the availability of skilled labour to put solutions in place.By observation, corporate IT departments have fallen in love with large multi-million dollar projects. It is almost as if a project has to cost north of a million dollars to be considered credible and important enough to warrant their time and attention. We believe the per user SaaS model has the ability to allow corporate IT to deliver massive impact to a business while spending a single digit percentage of these large projects. SolveXia is one of many vendors that now offer per-client per-month pay-as-you-go terms, which allows large projects to be delivered for less than $100k. The point here is that effective solutions don’t have to be expensive solutions.

    With respect to the availability of skilled labour and the timeline – this is a serious chokepoint in the delivery for most IT projects. Traditional big system implementations require the IT department to assemble (and pay for) a team of experts to work for 3,6 and 12+ months. They need to convert the user needs into designs, then to code the solution (although they often call it ‘configuring’ to make it sound less expensive). At the end, you get a system that hopefully bears a resemblance to what you needed 3,6 or 12+ months ago. We believe a better way is to put general purpose tool in the hands of the business people who need the systems, give them some initial training, then let them build, modify and run the processes they need. Marketing people call this ‘empowerment’. The key is to ensure that this self-service platform has the security, audit, documentation and other controls embedded into it.

    We do not have the systems in place to secure our sensitive data Excel is routinely used to store client and financial data – and organisations are rightly concerned about how to protect this information. It is well known in security circles that the single greatest risk in this case is that these files will be stored on laptops, emailed or copied onto USB keys that are then taken home on the train, and potentially lost or stolen.

    Some organisations try to address this by apply operating system profiles that require any removable storage devices to use password protection technologies (like BitLocker in windows).

    At SolveXia we try to address this issue by trying to avoid USB keys, email and local copies of files in the first place. By storing all the business process file in a web based single repository – any staff person who needs access to a file can sign in from wherever they (provided they are authorised to do so). Of course this web repository has to be secure. We used multiple layers of encryption on the data – both while it is moving and while it is at rest. We do not support unencrypted traffic to our platform. Most importantly, we work with the security teams within our clients to review the security profile of our infrastructure to ensure that it is at least as good as anything they have internally. The key point here is to provide people with a way of sharing information securely without resorting to high risk techniques.

So what does this all mean?

Let me summarise the key points we have covered about Excel Hell.

  • Excel Hell is widespread. Nearly every organisation suffers from it to some degree.
  • Excel Hell exposes an organisation to a wide variety of risk. Most of these risks are ‘hidden’ so they may not be top of mind.
  • It is short sighted to see Excel as the root cause of the problem. The root causes of Excel Hell generally point to end user requirements that are not, or cannot be, met by the corporate IT systems that are in place.
  • There are strategies available that allow an organisation to embrace all the (significant) benefits of having their staff utilise Excel, while actively managing the weaknesses and risks this introduces.
  • The corporate IT department has the ability to steer an organisation out of Excel Hell without spending large amounts of time or money. There are SaaS technologies and platforms available in the market today that can do this.

If you have a perspective on Excel Hell – please share your thoughts. You can reach me at james.simpson@sovlexia.com or using the commenting facility below.

Recent Posts
Showing 6 comments
  • Viral Panchal
    Reply

    It is true that “Excel Hell” makes organisations and subject matter experts victims, because they don’t manage, control or audit Excel based processes frequently.

    Given how important a role Excel plays in many organisations, I am surprised that they are not more eager to overcome Excel Hell by reviewing traditional processes of using Excel and using readily availble technologies to improve the situation. Is it corproate laziness? Is it that they are unaware of these technologies? Do they just not care to solve this problem?

    Thank you James for this article.

  • Nick James
    Reply

    This type of article is great at raising the awareness of the excel risk that most companies are either ignorant of or effectively ‘taking a punt’ that the data they produce is accurate. The reality is that even large corporates and well known companies struggle with mitigating the risks of using excel as a management tool.

    My thoughts are that this is not solely an IT issue, which is how many companies view the risk, it is a C level management problem and therefore should be treated as a core business challenge, the way to overcome this issue is to use the best available technologies such as SolveXia to effective eliminate and
    /or manage the excel risk (hell)

    • James
      Reply

      I think you are correct Nick with the point that this is a business challenge. In most of the businesses I deal with, because it involves Excel, it is automatically referred to and treated as purely an IT problem. Consequently, it does not get the priority or support required to solve it. Until is it is seen as a business issue (with very real business risks) then it will remain unsolved in the majority of organisations. Interestly, there are some (albeit only a few that I am aware of) that are dealing with it – and they see it is a way of building competitive advantage because they know that their competitors are mired in the problem.

  • Roman
    Reply

    Good article, I especially agree with establishing the ‘manage it’ vs ‘fight it’ mindset. Excel is a powerful tool, it is easily available, versatile and cost effective, it makes sense to take full advantage of it. Yes, having your business rely on Excel poses risks, but fighting Excel is a costly and counterproductive effort and really quite a futile one in this day and age. Given the amount of resources necessary to rid a business of Excel, and also given the subsequent inevitable productivity dip, managers are far wiser to take just a small fraction of those resources and invest it into a platform that enables secure Excel management and lets your organization enjoy the benefits of Excel, while all risks are eliminated or kept under tight control.

  • Brian
    Reply

    I’m one of the Excel addicts that uses the application in all three of the categories your article outlines. It is by far my most used productivity tool. So I might be termed an Excel demon if we continue the Excel Hell metaphor…
    The issues I face are more aligend to the manual aspects of bringing data from multiple sources into a single view – one I can then manipulate to suit my immediate requirements. These requirements may change very rapidly too of course – so I need and demand the flexibility to serve myself.
    So – if we accept the fact that I’m going to keep on doing it – the issue is more about the manual nature, the time it takes and the potential for error or disparity.
    Anything that takes away these issues but leaves me in peace to pivot to my hearts content has got to be a good thing.

    • James
      Reply

      I think you are safely in the majority of business users when it comes to how you use Excel – however I think you might be in the minority when it comes to recognising that you want to keep using this tool and address the specific weaknesses.

Leave a Comment