Death to Excel! Long Live Excel!
[/custom_frame_left] The widespread use of Excel in every part of business has created a phenomenon called ‘Excel Hell’. This is the confusion and frustration of not knowing which file is the right version to use, not being confident that it is secure and not having a sense of control over how it is used. In this article we look at why ‘Excel Hell’ happens, what the root causes are – and what can be done about it.
Automating business processes in 95% of organisations involves working with data in an Excel format. When you talk to enough businesses about automating processes, you inevitably find yourself having the famous ‘Excel Hell’ conversation. This involves hearing about the pain, frustration, fear, despair, anger and confusion of the IT department and/or technology director when it comes to managing the organisation’s Excel data.
Admittedly, having worked for Microsoft for 12 years, I was exposed to more than the usual share of these conversations. It is usually the IT Director or head of technology who speaks with the most venom about Excel. I find this puzzling because, if you had to list the most potent enabling desktop software technologies, surely Excel would feature in that list – and technology groups admire technologies that enable people right?
Well, in this case – no, they don’t. Here is a small sample of the quotes that capture the emotion surrounding this topic:
Excel is terrible for IT standards because staff keep creating their own spreadsheets to analyse and report with
We probably have more work taking place in Excel than in all of our sanctioned corporate IT systems combined
We have an IT strategy to remove Excel from the organisation – we will be spreadsheet free in 2 years
Excel is everywhere – every part of our business runs on Excel in some way or another – but we can’t keep track of it
and my favourite because of how much it unintentionally reveals:
We don’t like Excel because it moves too much decision making from the IT department to the end user
I find it fascinating that one of the most useful, widespread and impactful software technologies of the last 25 years generates this sort of commentary. It is also when I consider that working constructively with Excel is essential when automating business processes. SolveXia is in the business of providing organisations with a way to automate processes – so I thought it worthwhile digging deeper to discover why Excel Hell happens, and what can be done about it?
Why does Excel Hell happen ?
If it causes so much pain, why is Excel so ubiquitous? It cannot be just because it is installed on most corporate desktops. Availability alone does not auto-convert into usage. There must be reasons why people pick up Excel and do so much with it. I asked quite a few people, and this is what I heard:
- I can do things in Excel that our corporate systems can’t do
- I need my information structure in a certain way – and Excel is the fastest/cheapest way for me to get it like that
- I need my data from different corporate systems integrated – and I can get this done in Excel
- I can do ‘what-if’ analysis using Excel quickly and cheaply
- I can get some of my IT needs met without having to engage the IT department if I do it in Excel
In summary, it seems people use Excel to complement the systems and services that the IT department ordinarily provides. They can do things in a ‘self-service’ fashion which is faster and easier for them, and people like autonomy. This explains why people use it, and the fact that so many roles within an organisation nowadays are working with information in various states of readiness explains why so many people use it.
So what are people really saying ?
As with solving most problems, you need to look and listen for the real issues that are driving the human (and often emotional) response. That is, look beyond the symptoms, and search for the root cause. These expressions of frustration stem from more substantive, and very real, problems. It should be possible to build a “phrase translator” that summarises what people say and what people mean … and this is what I have so far.
|What is said||What is meant|
|There are too many spreadsheets in our organisation and we never know which is the ‘source of truth’||We do not have a way to track and version our important files|
|People keep critically important Excel spreadsheets on the C drive and the business never knows where this data really is||We do not have an effective approach to centrally storing and sharing our most important spreadsheets|
|People keep creating new spreadsheets and we have no control over the quality and correctness of the mini-applications that they build in Excel||We do not have an effective approach to controlling the quality of the applications and processes that run our business|
|People are doing work in Excel that should be done in our core systems||Our core systems are harder to work with than Excel, so people avoid them, preferring to work with what they feel makes them more productive|
|People use low quality cobbled together spreadsheet applications and processes instead of engaging IT and getting an enterprise quality solution||IT is either too expensive, to slow, or too hard for our staff to engage as means of fulfilling their business needs|
|We have sensitive data sitting in Excel that is not secured or encrypted as it just sits on people’s PC’s in directories and emails – this poses a serious security risk||We do not have the systems in place to secure our sensitive data|
|We have sensitive data Excel that is being shared between team members using USB keys and email – which can be lost or sent to the wrong person accidently – this poses a serious security risk||We do not have the systems in place to share our sensitive data between staff members securely|
Is there a way to deal with Excel Hell?
Yes. We have seen a number of our clients adopt strategies and tactics that manage these issues. Furthermore, we make addressing these issues an integral part of how we design our service – so that we can actively help our clients avoid or escape Excel Hell.
- Adopt a ‘manage it’ instead of a ‘fight it’ mindset. Some organisations, and some people, are inclined to fight the widespread use of Excel. This seems short sighted because it discounts the significant benefits Excel can offer. An alternative mindset is to seek to manage spreadsheets more actively, trying to put in place systems that retain the good things about Excel, while simultaneously reducing the impact of the bad. This is really a difference in attitude and approach more than a particular solution. When you speak to business and IT executives, you can tell pretty quickly if they are in the ‘fight it’ or ‘manage it’ mindset. The later tends to be more constructive.
- Think of Excel as three different products – and seek solution strategies for each. One way of thinking about Excel is that it is really 3 different applications/solution in one: It is used for (a) data storage (b) performing calculations and analysis, and (c) reporting and presenting information.Dissecting your organisation’s use of Excel along these lines can be helpful when you try to find ways to manage and mitigate the problems Excel Hell introduces. For example, by identifying those spreadsheets that are acting as data stores, you automatically highlight the set that most needs to be integrated and reconciled with established corporate data stores, and brought into version control. By identifying those spreadsheets that are primarily performing calculations and analysis, you have identified candidates for integration or inclusion in the corporate approach to application services. Identifying the spreadsheets that are primarily presentation and reporting, you have uncovered additional reporting requirements/components for the corporate reporting strategy.
Inevitably organisations have spreadsheets that perform 2 or all 3 of these functions. This warrants furthers analysis as Excel in this case has become a broader delivery platform – and it is wise for the IT department to understand the business needs that are driving this.
- Establish a central repository that has access and audit control for critical spreadsheets. One of the great advantages of Excel is that being a single file format, it can be versioned easily. There are plenty of inexpensive file versioning control systems in the market that are easy to setup and use. Beyond simple file versioning, there are ways to integrate Excel into a versioned and auditable process – while it continues to participate in business processes.
- Address the real substance of the frustrations. Most of our clients express some level of frustration with Excel Hell, and what I tell them is that instead of trying to ban or get rid of Excel, it is usually possible (and easier) to address the root cause of the frustration. Without meaning to make this article sounds like an advertisement, I will explain how we use the SolveXia platform we sell on a SaaS basis to complement existing corporate IT systems, and address the primary issues:
Problem How the SolveXia platform addresses this We do not have a way to track and version our important files The key here is to provide a single and simple system for storing Excel files that provides versioning. If this system is simple enough to use, and provides additional value around the Excel file, people will use it. We show clients that our platform is a simple, friendly and web based system that allows people to store their Excel files with versioning. They can always find the latest version of file, and be confident that it is the latest version. They can go back to prior versions if they need to – confident that the timeline of versioned files is accurate and reliable. We do not have an effective approach to centrally storing and sharing our most important spreadsheets We have chosen to address this issue in a very similar way to the issue above – provide a single simple system that people can store their important files in. The critical difference is that this is not like a LAN drive or file share (this approach will become a dumping ground). In the SolveXia platform everything is stored and organised in the context of a “business process”. You cannot just dump files randomly – they are attached to a modelled business process such as ‘produce end of week debtors balances’ or ‘reconcile invoices to the GL’. By requiring the storage of files in this way, we have found that sharing and communicating becomes more effective because it occurs at the business process level – which will nearly always include many files – rather than piecemeal file by file. We do not have an effective approach to controlling the quality of the applications and processes that run our business There are (at least) two aspects to this. When those responsible for process governance ask for control, they are asking for (1) complete information combined with (2) contextual information. Both of these need to be addressed in full if you going to deliver a sense of control and confidence.
In SolveXia we approached this request by building in what we call ‘automated documentation’ – the ability for our clients to click on a process and generate a single document that describes every step and every data item in the process – expressed in ‘plain English’. Technology has advanced sufficiently that this can be generated on demand, so it is guaranteed to be accurate. There is no risk of human error (writing down the wrong thing, or not writing down everything). As a result everyone see and review the logic that is contained within a process, and all the data sources it uses, and how it uses them. In this way the application (business process) can be reviewed with complete and contextual information.
Our core systems are harder to work with than Excel, so people avoid them, preferring to work with what they feel makes them more productive Fundamentally the only way to address this issue is to increase the priority and presence of self-service flexible applications. Most corporate IT systems have a very specific purpose (ERP, CRM etc…). They work well provided the business process stays within the understanding of the corporate system of how that process should be run. Inevitably however, that nature of business forces changes in process that test the limits of these large system. They were designed to be great specialists – not great generalists.We have tried to address this issue by providing clients with a system that is “general” in nature – that they can construct, modify and experiment with business processes with the same ease and speed that they can modify Excel spreadsheets. The key here is ensure that this general system has all the audit, transparency and security built into it. We believe that with the continuous pressure in business to do more with less, grow revenues and constrain costs – people will always look to experiment and innovate with the applications that support their business – you won’t be able to stop (and you probably don’t really want to). Rather than resist this, we focus on providing a platform that continues to deliver the flexibility of Excel but with the control attributes more typically associated with ERP, CR IT is either too expensive, to slow, or too hard for our staff to engage as means of fulfilling their business needs. Again there are (at least) two issues here. The first is cost and the second is the availability of skilled labour to put solutions in place.By observation, corporate IT departments have fallen in love with large multi-million dollar projects. It is almost as if a project has to cost north of a million dollars to be considered credible and important enough to warrant their time and attention. We believe the per user SaaS model has the ability to allow corporate IT to deliver massive impact to a business while spending a single digit percentage of these large projects. SolveXia is one of many vendors that now offer per-client per-month pay-as-you-go terms, which allows large projects to be delivered for less than $100k. The point here is that effective solutions don’t have to be expensive solutions.
With respect to the availability of skilled labour and the timeline – this is a serious chokepoint in the delivery for most IT projects. Traditional big system implementations require the IT department to assemble (and pay for) a team of experts to work for 3,6 and 12+ months. They need to convert the user needs into designs, then to code the solution (although they often call it ‘configuring’ to make it sound less expensive). At the end, you get a system that hopefully bears a resemblance to what you needed 3,6 or 12+ months ago. We believe a better way is to put general purpose tool in the hands of the business people who need the systems, give them some initial training, then let them build, modify and run the processes they need. Marketing people call this ‘empowerment’. The key is to ensure that this self-service platform has the security, audit, documentation and other controls embedded into it.
We do not have the systems in place to secure our sensitive data Excel is routinely used to store client and financial data – and organisations are rightly concerned about how to protect this information. It is well known in security circles that the single greatest risk in this case is that these files will be stored on laptops, emailed or copied onto USB keys that are then taken home on the train, and potentially lost or stolen.
Some organisations try to address this by apply operating system profiles that require any removable storage devices to use password protection technologies (like BitLocker in windows).
At SolveXia we try to address this issue by trying to avoid USB keys, email and local copies of files in the first place. By storing all the business process file in a web based single repository – any staff person who needs access to a file can sign in from wherever they (provided they are authorised to do so). Of course this web repository has to be secure. We used multiple layers of encryption on the data – both while it is moving and while it is at rest. We do not support unencrypted traffic to our platform. Most importantly, we work with the security teams within our clients to review the security profile of our infrastructure to ensure that it is at least as good as anything they have internally. The key point here is to provide people with a way of sharing information securely without resorting to high risk techniques.
So what does this all mean?
Let me summarise the key points we have covered about Excel Hell.
- Excel Hell is widespread. Nearly every organisation suffers from it to some degree.
- Excel Hell exposes an organisation to a wide variety of risk. Most of these risks are ‘hidden’ so they may not be top of mind.
- It is short sighted to see Excel as the root cause of the problem. The root causes of Excel Hell generally point to end user requirements that are not, or cannot be, met by the corporate IT systems that are in place.
- There are strategies available that allow an organisation to embrace all the (significant) benefits of having their staff utilise Excel, while actively managing the weaknesses and risks this introduces.
- The corporate IT department has the ability to steer an organisation out of Excel Hell without spending large amounts of time or money. There are SaaS technologies and platforms available in the market today that can do this.
If you have a perspective on Excel Hell – please share your thoughts. You can reach me at firstname.lastname@example.org or using the commenting facility below.