This blog site was hacked. Cracked. Whatever you want to say. We appear to have been hit by spammers / black hat SEO types. It turns out that we are not alone. So let’s talk about what happened and why.

First, though, if you use WordPress on your blog site and have NOT yet upgraded to WordPress, 2.5, STOP reading this article and go upgrade! We’ll still be here when you’re done.

WHAT HAPPENED?

In the last week of March, an attacker (or attackers) compromised our site and in a particularly insidious attack, added some text that was invisible to the viewer of our web pages, but was in the source of the file and therefore was seen by spiders from Google, Technorati and other search sites. If you looked at the pages of our site, you would not have a clue that they were the host to all sorts of spam links. However, if you looked at the source code, you would see something like this at the bottom of any given page (displayed as a screen capture so as not to give the spammers any more links):

Why was it “invisible”? Simple, the attackers simply added a ‘style=”none”‘ attribute to an HTML tag, in this case the good old <U> tag (underline):

That was it. Add a “style” attribute and in this era of stylesheet-aware browsers (which generally is a good thing), the text was invisible to the reader of the blogs.

WHY DID IT HAPPEN?

Fairly simple answer. We were still running a comparatively ancient version of WordPress… version 2.1.2. We had not upgraded to one of the more recent 2.3.x builds (although there are indications there are security issues with 2.3.3 as well (also here)). Yes, this is particularly embarrassing for us because we are a security organization but in a classic case of the “cobbler’s shoes” we were not staying up-to-date with the software here. VOIPSA is a volunteer, nonprofit organization and while that is not an excuse, that may explain it. There are a couple of us who do the system administration for VOIPSA’s site and we had discussed upgrading several times but given that we’d had some problems with earlier upgrades we wanted to have a block of time to do the upgrade – and we didn’t make that time. And as a result we were hit.

HOW DID IT HAPPEN?

It appears that the attackers used some type of PHP upload vulnerability to upload files to our site. We noticed a number of very small files that were uploaded which perhaps were their test files. We don’t (yet, anyway) know precisely what vulnerability the attackers exploited, but in looking at the changelogs for various versions of WordPress since the time of 2.1.2 it is very clear that several such vulnerabilities have been fixed in newer versions.

What the attacker(s) ultimately did, though, was to modify the “footer.php” file to include this “invisible” text above. Now where they did was a key factor. The first thing we did when we saw this was to check the ‘footer.php’ in the WordPress theme we are using for this blog. We use a very slightly modified version of the widely used Kubrick theme, primarily to bring in the graphic on the top, and it resides in its own directory. However, the modification was not there. We did some further searching and found the modifications in the “footer.php” file located in the “default” theme.

This puzzled us for a bit because we don’t use the “default” theme, but it would appear that the “wp_footer()” function must also call the footer.php file in the default directory as well as our own. I haven’t honestly crawled through the PHP code to figure this out, but by what we saw it would appear that this is the case.

Given that, the attacker’s task was fairly easy:

1. Find an older WordPress system with exploitable vulnerabilities.
2. Upload the modified file to “wp-content/themes/default/footer.php”

Ta da… invisible links are now there for sleazy SEO purposes.

HOW DID WE NOTICE IT?

Two ways. First, something the attackers did broke the web GUI editor. We still don’t know exactly what caused this (without going through all the WP code), but I knew something was wrong with the site when one of the other contributers contacted me to say he could not post a blog entry. You could go in and write the entry, but when you hit the “Publish” button you wound up with a completely blank screen. Something was causing the post.php file to fail. I didn’t notice this myself because I do all my writing offline (using MarsEdit) and that worked perfectly fine through the API. In looking into the problem with the editor, though, we saw the spammy links on the bottom of the posts and investigated further.

Second, the attackers seemed to get a bit cocky. They directly modified one of the blog entries. More than just the footer, they went in and modified one of the entries to include a couple of visible links. About the same time we were digging into the footer issue, one of the other contributors contacted us asking what was going on with all the spam links. Now it’s not clear that this was an exploit of the same vulnerability or of a different one, but it certainly clued us in to their being a problem.

WHAT HAVE WE DONE TO FIX IT?

First, we’ve upgraded to WordPress 2.5.

Second, we’ve gone through all the code in our themes (both the one we use and the default theme) ensuring there is no more bogus code (the upgrade seemed to take care of it all). We’ve also gone through our database to make sure there is no bogus text there.

Third, we’ve installed the “WordPress Automatic Upgrade” plugin which makes the WordPress upgrade process incredibly painless. Now that we have that plugin installed, it will be very easily to stay up-to-date with the very latest versions of WordPress. Note that it doesn’t automatically upgrade the site when a new version of WordPress comes out (we’re way too paranoid to allow that!) but it automates the tasks involved with an upgrade: backing up the themes and database, downloading the current version, de-activiating an re-activating plugins, etc. All the manual steps are now quickly done by the plugin.

NOTE TO WORDPRESS TEAM: Have you considered building this plugin directly into WordPress?

Fourth, we are giving a second look to hosted platforms (like WordPress.com) purely so that someone else can be responsible for system administration and upgrades – leaving us to just write. Being control freaks with a certain level of paranoia (as most security people are), we have resisted this in the past – and may still – but will be taking another look.

We are also suitably chastised (and angry) at having the site attacked so we’ll be having a renewed sense of vigilance with ensuring that this does not happen again.

FINAL THOUGHTS

The sad reality is that the spammers out there will always be looking for new and creative ways to game the system and do their dirty work. The reality is that they make money doing this for their clients and have every incentive to continue. The challenge we all have as operators of sites is to continue to try to stay ahead of the spammers – and to stay up-to-date with the software!

I would also just end with a word of thanks to all the WordPress developers who continue to fix security issues as they are found and who have made it such a great platform for blogging. I’d also thank the folks at Technorati who have made use of their massive database of blogs to notify people with vulnerable WordPress blogs (I received several email notices this morning) and who have stopped indexing afflicted blogs (denying the spammers a few of their links.)

Finally, I would encourage folks to read this article and it’s update over on the blog site Deep Jive Interests as they get into more about these attacks that have been going on across the blogosphere.

P.S. And if you haven’t upgraded to WordPress 2.5 yet, why not?

Technorati Tags:
wordpress, security, wordpress security, wordpress hacks, voipsa