This site is now updated to use WordPress 2.7, the newest version of WordPress. Everything seemed to go okay but if you do see any issues with the site please let me know via a comment here or via email.
For those of you who may be used to reading this blog through the “Security Bloggers Network” set up originally by Alan Shimel, you need to be aware that the “SBN” is going through a transition. As Alan details on his blog, Google is in the process of shutting down the “Network” feature of Feedburner and as a result the page and feed for the SBN will be going away.
Alan is working on a new solution but in the meantime you may want to grab the OPML file for the Security Bloggers Network (you should then be able to import this into most feed readers). There are a lot of great security blogs out there.
Stay tuned for more information – once Alan has another solution in place I’ll post an update.
We’ve upgraded this site to WordPress 2.6.2… there shouldn’t be any issues with the site, but please let us know if any pages act strange. Thanks, Dan
This blog site was hacked. Cracked. Whatever you want to say. We appear to have been hit by spammers / black hat SEO types. It turns out that we are not alone. So let’s talk about what happened and why.
First, though, if you use WordPress on your blog site and have NOT yet upgraded to WordPress, 2.5, STOP reading this article and go upgrade! We’ll still be here when you’re done.
In the last week of March, an attacker (or attackers) compromised our site and in a particularly insidious attack, added some text that was invisible to the viewer of our web pages, but was in the source of the file and therefore was seen by spiders from Google, Technorati and other search sites. If you looked at the pages of our site, you would not have a clue that they were the host to all sorts of spam links. However, if you looked at the source code, you would see something like this at the bottom of any given page (displayed as a screen capture so as not to give the spammers any more links):
Why was it “invisible”? Simple, the attackers simply added a ‘style=”none”‘ attribute to an HTML tag, in this case the good old <U> tag (underline):
That was it. Add a “style” attribute and in this era of stylesheet-aware browsers (which generally is a good thing), the text was invisible to the reader of the blogs.
WHY DID IT HAPPEN?
Fairly simple answer. We were still running a comparatively ancient version of WordPress… version 2.1.2. We had not upgraded to one of the more recent 2.3.x builds (although there are indications there are security issues with 2.3.3 as well (also here)). Yes, this is particularly embarrassing for us because we are a security organization but in a classic case of the “cobbler’s shoes” we were not staying up-to-date with the software here. VOIPSA is a volunteer, nonprofit organization and while that is not an excuse, that may explain it. There are a couple of us who do the system administration for VOIPSA’s site and we had discussed upgrading several times but given that we’d had some problems with earlier upgrades we wanted to have a block of time to do the upgrade – and we didn’t make that time. And as a result we were hit.
HOW DID IT HAPPEN?
It appears that the attackers used some type of PHP upload vulnerability to upload files to our site. We noticed a number of very small files that were uploaded which perhaps were their test files. We don’t (yet, anyway) know precisely what vulnerability the attackers exploited, but in looking at the changelogs for various versions of WordPress since the time of 2.1.2 it is very clear that several such vulnerabilities have been fixed in newer versions.
What the attacker(s) ultimately did, though, was to modify the “footer.php” file to include this “invisible” text above. Now where they did was a key factor. The first thing we did when we saw this was to check the ‘footer.php’ in the WordPress theme we are using for this blog. We use a very slightly modified version of the widely used Kubrick theme, primarily to bring in the graphic on the top, and it resides in its own directory. However, the modification was not there. We did some further searching and found the modifications in the “footer.php” file located in the “default” theme.
This puzzled us for a bit because we don’t use the “default” theme, but it would appear that the “wp_footer()” function must also call the footer.php file in the default directory as well as our own. I haven’t honestly crawled through the PHP code to figure this out, but by what we saw it would appear that this is the case.
Given that, the attacker’s task was fairly easy:
- Find an older WordPress system with exploitable vulnerabilities.
- Upload the modified file to “wp-content/themes/default/footer.php”
Ta da… invisible links are now there for sleazy SEO purposes.
HOW DID WE NOTICE IT?
Two ways. First, something the attackers did broke the web GUI editor. We still don’t know exactly what caused this (without going through all the WP code), but I knew something was wrong with the site when one of the other contributers contacted me to say he could not post a blog entry. You could go in and write the entry, but when you hit the “Publish” button you wound up with a completely blank screen. Something was causing the post.php file to fail. I didn’t notice this myself because I do all my writing offline (using MarsEdit) and that worked perfectly fine through the API. In looking into the problem with the editor, though, we saw the spammy links on the bottom of the posts and investigated further.
Second, the attackers seemed to get a bit cocky. They directly modified one of the blog entries. More than just the footer, they went in and modified one of the entries to include a couple of visible links. About the same time we were digging into the footer issue, one of the other contributors contacted us asking what was going on with all the spam links. Now it’s not clear that this was an exploit of the same vulnerability or of a different one, but it certainly clued us in to their being a problem.
WHAT HAVE WE DONE TO FIX IT?
First, we’ve upgraded to WordPress 2.5.
Second, we’ve gone through all the code in our themes (both the one we use and the default theme) ensuring there is no more bogus code (the upgrade seemed to take care of it all). We’ve also gone through our database to make sure there is no bogus text there.
Third, we’ve installed the “WordPress Automatic Upgrade” plugin which makes the WordPress upgrade process incredibly painless. Now that we have that plugin installed, it will be very easily to stay up-to-date with the very latest versions of WordPress. Note that it doesn’t automatically upgrade the site when a new version of WordPress comes out (we’re way too paranoid to allow that!) but it automates the tasks involved with an upgrade: backing up the themes and database, downloading the current version, de-activiating an re-activating plugins, etc. All the manual steps are now quickly done by the plugin.
NOTE TO WORDPRESS TEAM: Have you considered building this plugin directly into WordPress?
Fourth, we are giving a second look to hosted platforms (like WordPress.com) purely so that someone else can be responsible for system administration and upgrades – leaving us to just write. Being control freaks with a certain level of paranoia (as most security people are), we have resisted this in the past – and may still – but will be taking another look.
We are also suitably chastised (and angry) at having the site attacked so we’ll be having a renewed sense of vigilance with ensuring that this does not happen again.
The sad reality is that the spammers out there will always be looking for new and creative ways to game the system and do their dirty work. The reality is that they make money doing this for their clients and have every incentive to continue. The challenge we all have as operators of sites is to continue to try to stay ahead of the spammers – and to stay up-to-date with the software!
I would also just end with a word of thanks to all the WordPress developers who continue to fix security issues as they are found and who have made it such a great platform for blogging. I’d also thank the folks at Technorati who have made use of their massive database of blogs to notify people with vulnerable WordPress blogs (I received several email notices this morning) and who have stopped indexing afflicted blogs (denying the spammers a few of their links.)
Finally, I would encourage folks to read this article and it’s update over on the blog site Deep Jive Interests as they get into more about these attacks that have been going on across the blogosphere.
P.S. And if you haven’t upgraded to WordPress 2.5 yet, why not?
FYI, this site is now running WordPress 2.5 and everything seems to be fine. If you experience any issues commenting on posts or see anything strange about how the site appears, please do let us know.
We know return you to your regularly-scheduled VoIP security news…
P.S. I should note that we’ve now installed the “WordPress Automatic Upgrades” plugin and it worked very well to upgrade this site to WP 2.5. We’re not adventurous enough to let it just automatically upgrade the site by itself, but we’ll certainly use it to manually upgrade the site in the future.
Just FYI, we’re going to be upgrading this site to WordPress 2.5 sometime in the next couple of days. There should be no noticeable impact to the site but if you land here sometime and the site looks a bit strange, it may be because we’re in the middle of the upgrade. (We do not expect to have any issues moving the theme we use to WP 2.5, but we don’t know.)
FYI, if any of you are incredibly attentive, you may notice that the URL for the RSS feed has changed on the site.Â You don’t have to do anything if you are currently subscribed.Â The old feed URL works just fine.Â We are just now promoting the Feedburner URL so that we can get a better view of stats around subscribers.Â The Feedburner feed is basically just a wrapper around the old feed URL, so the content is identical. (If you want to unsubscribe with the old URL and subscribe with the new one, you can, but no need to do so.)
Careful observers of this site may have noticed a new sidebar block titled “VOIPSA Member Blogs”.Â This is something we decided to do for members of the VOIPSA Technical Board of Advisors and other folks involved with VOIPSA.Â Please do check out those other weblogs, and if you are a TBA member with a blog who missed my original message about this sidebar, please do contact me and we’ll be glad to add your weblog.
A comment to one of Shawn’s recent posts made me realize that I hadn’t mentioned here that some time ago this blog was added to the “Security Bloggers Network“, which is a Feedburner “network” of blogs. If you follow the previous link, you’ll see links to the 68 blogs currently part of the network. You can read the blogs, subscribe to their individual feeds, or, if you want, you can subscribe to the network feed to get all posts across all network blogs. We glad to be a part of the network and getting news about VoIP security out to an even broader network. We encourage you to check out the network home page and explore many other great blogs out there on security.