the story
This is a small “disaster” summary on how one small mistake in an email setup lead to a company wide email outage and several entries on spam lists.
Consider the following scenario: small company, few imap accounts, one domain (we call worksfine.com) and one catchall/forward domain (we call fluky.com).
Some day, fluky.com is down again, so the forwards are temporarily not working. Terrified by that, the company calls another “admin”, as the usual guy is on vacation or otherwise unavailable.
The admin sees the problem in the imap accounts, which are tied to worksfine.com, not the real reason that fluky.com forwards are currently simply not forwarding. So he switches the accounts on a few computers from imap to pop3 (fetch and delete from server). Of course the problem persists, but magically vanishes after a while. Yeah, you guessed it, fluky.com is up again and delivering the mails which accumulated during the downtime.
Now comes the interesting part. On of the workers on a changed workstation sends an email, with a 10mb attachement. The workstation is using a nice MS product, sends the mail fine but fails to move it to the sent folder and puts it back into the send queue for a retry. I cannot exactly explain why this has happened, as I did not yet have time to inspect the machine. From what I know I guess, that the sent folder was still the imap account, but the mail server addresses for pop are different so the ssl certificates did not match and the move failed. Or some even simpler reason. ( insert Murphy’s Law here)
Luckily, that email was addressed to the whole company and a handful of customers. So after a few minutes the DOS “attack” on the own company was successful. Also a few customers were slightly upset and the address did end up on the blacklist filters on mail appliances and mail providers used by the customers.
lessons learned
- murphy’s law, hanlon’s razor, etc
- use remote access software and set it up in advance. (the initial analysis was via phone, to a non native speaker in a different country, using a different language version of the OS and was not technically skilled)
- deny any unwanted access or usage if somehow possible
- provide a summary of the systems and how they interoperate in case extra personal is brought into the company without prior notice. And put a big DO NOT list on it
- don’t panic 😉
Nico Heid
Latest posts by Nico Heid (see all)
- Raspberry Pi Supply Switch, start and shut down your Pi like your PC - August 24, 2015
- Infinite House of Pancakes – code jam 2015 - July 13, 2015
- google code jam 2014 – magic trick - April 18, 2014