A Blackhole of E-mails

A look into a tech postmortem write-up

In an infamous Reddit post from r/talesfromtechsupport, a unique tech issue plagued the email servers of an unidentified academic institution in 2004.

Issue Summary

With the intention of needing an email address created for the purpose of updating all affiliated emails on the goings on of an institution, an email address was created (ex. all-employees@org.edu). During daily operations, a user became aware of the address and used it to send a message inquiring about a local apartment rental to all 30,000 user emails. This immediately triggered a mass influx of messages from a myriad of sources, the first being individuals replying to the message asking why this message was being sent their way. Once this proceeded, this prompted its own wave of messages replying all to the responses, confused as to what the origin of the messages even were. Lastly, users found the largest wave of messages in the form of auto-replies. In an organization with tens of thousands of people, it just so happened about 20 or so people were on vacation and set an away message to auto-reply to incoming messages. This prompted an away response to be sent for every single message received within the mailbox, including responding to the other away messages received. With messages compounding, the mail servers shortly went offline, unable to handle the capacity.

Timeline

From the point of the offending email being sent and the server coming back up, it took a little over 24 hours to restore the mail servers.

Root Cause

When the all-employees@org.edu email was created, it was left with permissions unset, so any user could use it for a message. Not only was it supposed to be available for a small handful of people to use to send emails, it was also supposed to have the reply all function disabled to avoid mass email threads.

Corrective and Preventative Measures

In order to correct the damage, the IT department had to go through each individual mailbox and purge the massive amount of messages sent from the chaos. Once this purge was done, the auto-replying away responses had to be disabled and the functionality was modified to deter auto-replies from responding to other auto-replies. The IT department also made sure to modify the all-employees@org.edu email to only be able to be used by certain users and set permissions to avoid a user replying all on an email from that address.

— — Written by Kathleen R McKiernan for Holberton School NHV — —