I’ve gone trough some changes with Ubuntu in the year. I started with Xubuntu 10.04. In time, I moved away from XFCE to KDE then then quickly to Gnome. Somewhere in between I also upgraded to 10.10, but I’m not sure about the chronology. With Ubuntu 11.04 came Unity, which I have a fair chance for about 2 weeks. Not really liking it, I have Gnome 3 / Gnome Shell a try. Anybody who gave Gnome Shell a try knows its not anywhere near as customisable as Gnome 2 was. That’s okay they, ’cause it rocks as-is.

The website in question is a news website. Its content is basically text+images. The text, obviously, was in the database. The images where, obviously, on the filesystem. So logic would suggest that transferring the files and database would do the trick, right? (Ignore the DNS for a moment, okay?)

And sure, normally it would. And all the files where there, except for the images. Taking a closer look, it turned out a part of the images (the first 1998 to be exact) where. in fact, there. Taking an even closer look made me notice that a part of the response text when doing an FTP LIST of the images directory was “TRUNCATING RESULT AT 2000 MATCHES”. Pretty fucking awsome, right, they limited their FTP LIST to 2000 items (‘.’ and ‘..’ being 1 and 2).

I called the support phonenumber, but they couldn’t help me – apparently increasing that 2000 would be bad for server performance. They suggested I grab 2000 files, delete them on the server and then get the next 2000. Nice idea, except that’d mean the website would be broken while I’m getting the files. (The actual switch will be monday/thuesday, so removing all but 2000 images now is not an option.)

Long live FTP access though – I wrote a small PHP script that would list (links to) all the files in the directory, install the firefox plugin ‘download them all’, and about two hours later, I had all the files sitting on my harddrive nicely. (And I’m sure the 14.000 HTTP requests where a much bigger hit on their server then allowing me to grab them trough FTP would have ever been.)

So, in the end, this was just a dumb limitation that (in this specific case) did more harm then good.

Since we’re working with kids, and there are always kids that try to screw something up, we want to be able to stop that kid, before we’re kicked off of the servers by the people who manage things world wide. (Which can happen if you spam to much, swear, that type of thing.) But to stop the kid, we need to know he’s screwing up. And with 30 kids at a time and just 4 of us, you can’t do that unless you can centralise everything.

So, thats where I came in. In a few iterations over the years, we started with a simple C program that dumped all the outgoing IRC messages to a console, and ended up with a Java program that scans the outgoing IRC messages, nickname changes, etc. on our gateway server, gives it a spam score based on some criteria, and stores it in a MySQL database on a different server. The second server is a simple LAMP (Fedora to be exact), plus that is has two screens on which we see the newest outgoing messages, colour coded based on the spam score that was given. This happens on a different server, because in our situation (with a budget of €100~€150 a year, mostly relying on donations of hardware) our internet gateway server isn’t the most powerfull machine, and we can’t get a more powerfull machine, but we can get another not so powerfull machine. So different things get different servers. (These are SRS1 (gateway) and SRS3 (LAMP) on the diagram.)

So, that was last year. Everything worked without a problem, traffic logging happened the way we wanted it to, and it proved to be a very usefull tool for us. This year, the guys wanted to change some things. To see if they didn’t break anything, they wanted to first build up the network as it existed last year. Everything looked like it worked, but as soon as the Java program needed to log something, it crashed with a database connection exception. They double-checked the config, they tried again, and it crashed again. Then, they called me. Since I didn’t have anything better to do (this was last night) and I wanted to head over to where they where anyway, as they where going to a bar afterwards, I came over to see what was wrong.

So, I took a look. The Java program was running correctly, it was indeed an issue connecting to the database. Going to that server, the database was running. The output was working properly, I could look into the database using phpMyAdmin, everything was fine. The configuration for the Java program was fine, as well. In fact, nothing had changed, except for one thing: The harddisk of SRS3 was moved to a new system that was much smaller and more powerfull. But the system was running without an issue.

Oh, yea, and it installed some updates. Well, if you start a system for the first time in about 10 months, you can expect a few updates. What we didn’t expect, was that one of those updates would cause the firewall to stop letting traffic trough on port 3306 (which MySQL uses). Problem solved. (And luckily it wasn’t somewhere in a production environment – not that I’d use Fedora there.)

Bah, this was supposed to be short.

And people say Linux isn’t user friendly.