Fnord.

This morning, in my inbox, this is what I saw:

That is, 37 reports that a host had tried to break in to my SSH port, and failed more than one password attempt. If I weren’t running DenyHosts, the dictionary attacks would have a lot greater chance of succeeding. As it is, I’ve seen at least 50 hosts blocked today; usually I get one every few days. Sadly, I’ve not got over sixty five thousand hosts denied via DenyHosts, which is a sad commentary on how many cracked Windows machines are out there.

Patch and monitor those boxes, people!

Today I found a link from Project Honeypot to the http:BL plugin for WordPress that blocks various forms of crap - comment spam being a big deal here, as I somehow managed to accumulate over 108 thousand of the attempts on this no-name blog.

Yeesh. Thank god for Akismet.

So the plugin is running, please send me email if you see any problems from it. It’ll be interesting to see what it does to my visit stats, I suspect many of the ‘visitors’ are spam bots anyway.

We were down last night from about 11:30PM until today around 11AM. Somehow, some [censored] managed to add a link to my sidebar by inserting into the wp_links table for wordpress.

After thinking about it for a bit, I’m updating this post: The link was to www.snjpc.com. It’s an online casino site, and while I dislike giving them hits it seems worthwhile to say that they, or their hires, are doing this sort of blog cracking.

How they managed this, I am still trying to determine. The mysql logfiles

cd /var/log/mysql
mysqlbinlog *.0* | grep wp_links

show no inserts, the mysql instance is only available from localhost (firewalled to boot), and system logs look good. Checks for rootkits and similar are negative, firewall logs are clean, and I am puzzled. Props to this page for mysql tips.

One piece of evidence - they seem to have borked wordpress, probably on purpose - if you invoke the theme editor (or plugin editor) you get

Sorry, that file cannot be edited.

which is patently false. My working hypothesis is, for now, that someone managed to either inject an exploit into Debian’s wordpress package, or found a generic Wordpress exploit. I am continuing to investigate and take various measures to prevent a reoccurence. This sucks.

Oh yeah, swapped themes for a while as part of the cleanup and diagnostics. Let me know what you think.

Update: Added paragraph naming snjpc.com

Update: A fresh download of Wordpress 2.2.2 from Debian or wordpress.org produces the same result… interesting. So either the default WP is broken, or its a config file/database problem. Debugging continues…

Update: Looks like the problem is related to the list of files passed to the validate_file_to_edit function. It looks like the first file passed is the wp-config.php, which on Debian is a link to the system file in /etc/wordpress. Hmm, maybe that’s how they got the MySQL access information… You can hack around this by adding the following to theme-editor.php. Change this:

if (empty($file)) {
$file = $allowed_files[0];
}
$file = validate_file_to_edit($file, $allowed_files);

to the following. This overrides the file list that was passed in, so you have to change the index to match the file you want. Other than that, it totally fixes the problem. Hmm…

if (empty($file)) {
$file = $allowed_files[0];
}

$file = $allowed_files[3];

$file = validate_file_to_edit($file, $allowed_files);

From the new firewall’s syslog:

Wed Jun  6 17:26:05 2007
RGFW-IN: BLOCK-SYNFLOOD (TCP 76.171.170.53:34556->204.128.136.1:80 on ixp1) [200,0]
Wed Jun  6 17:19:22 2007
SYSLOG_NK-(System Log)Mail sending to [redacted] successfully !

Seems like the box think’s there’s a SYN flood attack, which is a particular type of denial of service.

False alarm? Occurring all along and just now visible? Can’t tell. Does make me wonder if my old OpenBSD pf rules were also working on these. pf is amazing for firewalls.

(The IP points to a host on the Road Runner cable modem network, I assume its a pwned Windows box.

usul:/etc# wc -l hosts.deny
29836 hosts.deny

Holy cow! All this since May 14th 2006.

Non-nerd translation: If you try to login to my server via SSH, Denyhosts adds your hostname to the block list after a certain number of bad usernames or passwords. Since May 2006, there have been 29,836 such hosts added to the blocklist. All are botnet-driven Windows boxes, trying dictionary and default passwords, but its still pretty annoying. I wonder how well tcpwrappers, which has to parse this file for every connection, scales?

Sigh. They try to spam the site via the email signup. So I can remember, the magic is:

mysql -u root -p
use wordpress;
delete from wp_email_list where gets_mail is NULL;

2031 records. Ye gods.

I hit breaking point yesterday when I realized that I’d gotten 2668 spam in about three weeks. The main culprit is my old computer.org email address, which I’ve had since ‘95 or so, and the IEEE spam filters are ineffective.

My phfactor.net address is also discovered, leading to much annoyance. Lately its been stock pump-and-dump scams, delivered as images. The Apple mail.app filtering is excellent, but can’t deal with image-delived spam.

So yesterday I signed up for Spamcop:

They run a clean setup - you forward your email to them, and they have POP/IMAP/WWW access to it. Costs $30 per year, with discounts for multiple addresses. So far it’s been 100% effective, even catching the image spam. If you look at the headers, you can see they run Qmail (Hi Dan!).

Anyway. I’ve updated all my old addresses to use this now (IEEE, Computer.org, fnal.gov, anl.gov, mcs.anl.gov, phfactor.net) so we’ll see how it works. If it remains effective, it’ll totally be worth every penny.