Every year, there is the JOTI (See http://www.joti.org/en/index.html and http://jota-joti.scoutingrijen.nl/en/home ) event. We (Scouting Rijen, the local Scouting branch) join the event every year. The last 4 years, I was a part of the team organising that. (But I stopped after last year, as the work that I find interesting is all done.) Simple put, we have about 30 workstations, and a bunch of systems that act as servers for one thing or another. (Diagram)
Since we’re working with kids, and there are always kids that try to screw something up, we want to be able to stop that kid, before we’re kicked off of the servers by the people who manage things world wide. (Which can happen if you spam to much, swear, that type of thing.) But to stop the kid, we need to know he’s screwing up. And with 30 kids at a time and just 4 of us, you can’t do that unless you can centralise everything.
So, thats where I came in. In a few iterations over the years, we started with a simple C program that dumped all the outgoing IRC messages to a console, and ended up with a Java program that scans the outgoing IRC messages, nickname changes, etc. on our gateway server, gives it a spam score based on some criteria, and stores it in a MySQL database on a different server. The second server is a simple LAMP (Fedora to be exact), plus that is has two screens on which we see the newest outgoing messages, colour coded based on the spam score that was given. This happens on a different server, because in our situation (with a budget of €100~€150 a year, mostly relying on donations of hardware) our internet gateway server isn’t the most powerfull machine, and we can’t get a more powerfull machine, but we can get another not so powerfull machine. So different things get different servers. (These are SRS1 (gateway) and SRS3 (LAMP) on the diagram.)
So, that was last year. Everything worked without a problem, traffic logging happened the way we wanted it to, and it proved to be a very usefull tool for us. This year, the guys wanted to change some things. To see if they didn’t break anything, they wanted to first build up the network as it existed last year. Everything looked like it worked, but as soon as the Java program needed to log something, it crashed with a database connection exception. They double-checked the config, they tried again, and it crashed again. Then, they called me. Since I didn’t have anything better to do (this was last night) and I wanted to head over to where they where anyway, as they where going to a bar afterwards, I came over to see what was wrong.
So, I took a look. The Java program was running correctly, it was indeed an issue connecting to the database. Going to that server, the database was running. The output was working properly, I could look into the database using phpMyAdmin, everything was fine. The configuration for the Java program was fine, as well. In fact, nothing had changed, except for one thing: The harddisk of SRS3 was moved to a new system that was much smaller and more powerfull. But the system was running without an issue.
Oh, yea, and it installed some updates. Well, if you start a system for the first time in about 10 months, you can expect a few updates. What we didn’t expect, was that one of those updates would cause the firewall to stop letting traffic trough on port 3306 (which MySQL uses). Problem solved. 
(And luckily it wasn’t somewhere in a production environment – not that I’d use Fedora there.)
Bah, this was supposed to be short.