The worldwide outage from CrowdStrike exposes many issues that people need to talk about more. Over the recent years we have become more and more dependent on IT systems but as a society we have badly under-invested in security and quality assurance for those critical systems. This has led to comics like this one from KXCD (source https://xkcd.com/2347/):

The Crowdstrike outage effected a hug number of systems worldwide that were running the microsoft windows operating system. For the most part running critical infrastructure on the Windows operating system is a bad idea and incidents like this indicate why. The CrowdStrike software is an anti-virus package that runs on the Windows operating system. An automatic update to the virus definitions caused the systems to crash with a kernel panic. Kernel Panics, also known as the "blue screen of death", used to be far more common on Windows systems. In fairness to Microsoft their operating system stability is greatly improved these days compared to those early days. However it is slightly concerning that a non-malicious automated remote code deployment could cause a kernel panic.

I find it shocking that we still have such RCE problems on such important systems like this. Even more shocking is how these changes were not run in a staged manner. As far as I understand it these updates were applied everywhere all at once. Unfortunately important and even critical systems not being rolled out properly is all too common these days. The downtime and losses that occur from this sort of procedural negligence is just seen as a "cost of doing business" these days. Much like the widespread problems with data leaks the cost is mostly borne by the users and not the companies. This causes an extreme misalignment of incentives where it is more profitable for any individual company to under-invest in infrastructure reliability than it is to invest in it, at least in the short term. But in the long term everything turns into a complete mess for everyone and on a societal level this is a really bad outcome. This is the technological Tragedy of the Commons we see playing out in real time.

Thankfully most of the internet infrastructure these days is run on Unix/Linux systems so the damage here was not completely catastrophic. However the same can't be said for many other parts of the societal infrastructure, many Automated Teller Machines for example run on windows. Many businesses run their operations on computers that run Windows. As we saw from the wannacry ransomware incident many hospitals run their entire operations on windows machines. Just like what happened before in 2017 with Wannacry there were multiple hospitals that couldn't operate properly due to this outage. It is not a stretch at all to say that many people would have died as a direct result of this outage.

It goes to show if some other event were to happen that took out more machines or even the same amount of machine but for longer we would have significant economic impacts within days and complete collapse of certain organizations within weeks. It would be exceedingly costly on society to have a large outage like this if it went for anything more than a day or two.

Many organizations simply do not have backup processes for when these software systems fail. Some don't even have good enough processes to ensure continuity of service even when these underlying software systems don't fail. This is a large part of why IT outages are now so costly to society, many organizations literally have no backup plans.

I was talking to many people in my professional network and I remember one company had such a bad outage that they couldn't even use their printers. This company had made the horrible mistake of going with SAP software and was running their software on Windows (and they made this choice in recent years which is inexcusable since they made this choice after thew Lidl debacle). Their competitors were running vastly cheaper Shopify systems and experienced no downtime, more expensive systems are not necessarily better in the software world but yet the psychology of Veblen goods is still highly influential in the board room and hence is part of all corporate IT systems sales pitches. The infrastructure behind Shopify is a great example of the changes in the economics of IT systems in the modern era but that is a deep topic for another post.

Understanding the economics of IT system choices

I directly saw two businesses in the same industry, one chose SAP and one chose the vastly cheaper Shopify platform. With this outage the business using the cheaper Shopify was still up and running with no downtime while the business using SAP was facing major disruptions.

Part of the dangers of modern IT systems is that the greatly improved efficiency they enable, when things work, combined with long periods when things work, creates an extremely strong psychological situation where forgetting about backup systems becomes the norm over time. This is because barring certain exceptional cases (which would take an entire blog post to only scratch the surface on) the human mind strongly optimizes for reduced energy expenditure. Our minds will forget rare cases if they don't come up in a long time, they fall out of the realm of perception. Putting your entire business at risk from making IT systems choices might be irrational when viewed from a classical economics point of view that assumes people are not only rational economic actors with an infinite time horizon but that they are operating under perfect information availability when they are considering their options. But yet putting the company at risk via poor IT choices happens literally every day in many companies and organizations world wide. This behavior is so predictable that it highlights the failure of mainstream economics to account for people's actual behaviors. Whole books like the aptly named "Predictably irrational" by Dan Airely talk about this.

From an economic point of vew we have calculations about the Expected Value and Expected Utility of decisions made that take a weighting of how often and how big the impact is from certain things. Where many people have very large troubles in their natural mode of thinking is in assessing rare events. Low probability events that have very large wins and large losses are something that the untrained mind typically does poorly at assessing. The prevalence of whole industries based on sucker bets, like the lottery, is a perfect proof of this if you needed one.

On top of this we have the behavioral economics of standing your ground on defending decisions that are costly now, or in the immediate financial quarter, that will only pay off some unspecified time in the future. The incentives for a staff member who understands the risks are usually such that the staff member will make more money by keeping quiet than they will by speaking up. Even in jurisdictions that hold companies legally liable for certain mistakes that cause harm being a whistelblower is a very risky move for a staff member. Most staff will do a calculation from their own perspective and come to the conclusion that quietly quitting is much more profitable than rasing a scene. This effect is amplified in any culture that prioritizes consensus over competence since a disagreement about risks will be a challenge to the consensus thinking.

The dangers of a cashless society

A purely cashless society will have catastrophic failure modes from internet infrastructure outages.

In the past if your computer systems had outages you could still do things like buy food and fuel. In a cashless society all economic activity will halt almost instantly in the case of an internet outage that impacts payment systems.

We have seen a few of these over the last few years as cashless payments have become more common. A huge outage in Ontario Canada basically caused the economy to grind to a halt in a disturbingly short amount of time. Payment terminals were out of action, people couldn't buy fuel to get to work. Within hours the second order effects of people not being able to buy food and fuel were already being seen.

Given that we still have massive cyber outages we most certainly aren't ready to shift to digital currencies. We aren't even investing anywhere near enough in cyber security as a society to make our currency systems be safe in a purely digital system with no backup options. We are also not investing anywhere near enough in technological infrastructure to make this a safe move even if there were no cyber security concerns.