Alert Fatigue

alarm

My sons make quite a ruckus learning to play their musical instruments (a drum and electric guitar).  But there is another type of noise that can cause just as big a headache: digital noise.  Our electronics beep and chirp and ring, helping us keep pace with our hectic lives.  Our computers, phones, tablets, alarms, door bells, and watches are always vigilant.  They wake us in the morning, remind us of our next appointment, notify us when our friends post updates, let us know when a package was delivered, tell us when it’s time to leave, warn us of inclement weather, notify us that someone is calling or texting, etc.  Modern-day cars are just computers on wheels, with their own lights and beeping noises.  Even our appliances are getting in on the action.  Microwaves, ovens, washers, and even refrigerators have their own warning signals.

Recently I missed an appointment.  I had received a text message the day before and I had a reminder on my calendar.  I realized, however, that I have become immune to all these noises.  There are so many alerts vying for my attention that I have practically grown numb to their cacophony.

By mistakenly missing my appointment, I am reminded of an important lesson… the value of separating out the important from the mundane.  Farmers used wind to separate wheat from chaff.  Gold miners used a pan to differentiate gold from other deposits.  Banks use machines to rapidly sort real money from counterfeit bills.  Throughout history, people have come up with clever inventions to filter out noise and reduce “alert fatigue.”*

The National Institute of Health published a study on the impact of computer systems generating alerts for prescription drug interactions and concluded:

Clinicians became less likely to accept alerts as they received more of them, particularly repeated alerts.*

This principal is also true in technology.  It is very important to monitor our applications and infrastructure.  Ideally we identify an issue before it causes an outage.  When outages do occur, however, the right people need to know and address the issue as fast as possible.  Outages result in lost revenue.  Amazon had a 63 minute outage in 2018 that was estimated to have cost it almost $100 million.*  A 14 hour outage in March 2019 cost Facebook about $90 million.*  If alerts go off too often technicians just snooze them and the alarm becomes meaningless, resulting in increased downtime.

Monitoring tools have become great at identifying problems and triggering alerts.  Companies should monitor multiple aspects of their systems for issues that may cause outages or for signs of a security breach.  Computers can trigger alerts for just about anything from running low on disk space to overheating to bugs in software applications to unusual login activity.  Many alerts are really just notices that the computer is running some cleanup process or installing some patches.  If senior technicians are woken in the middle of the night for every issue they will quickly start to tune out the alerts.  It is crucial to automatically prioritize the alerts and only page IT staff for truly crucial problems.  A corollary to this is the need to page the right people and only the right people.  Otherwise companies end up with “it’s not my problem” syndrome and critical alerts will go unanswered.

Alert fatigue prolongs outages and impacts the bottom line.  It may also jeopardize security if the alert is for a potential data breach.  Tuning alerting systems is not always easy.  It requires good judgement and some trial and error.  Filter out too many alerts and you might miss something critical.  Filter out too few alerts and you end up with alert fatigue.  It is just like screening your phone calls.  If you don’t screen the calls you end up on the phone with a pushy salesperson.  But if you screen too many calls you end up missing someone you really need to speak with.