Voice of VOIPSA

Our apologies for the outage of both this blog and the main VOIPSA web site over the last weekend – and many thanks to all of you who wrote in to let us know. We recently moved the site to a new hosting provider and unfortunately it seems that in the initial move they missed moving over the domain name. That has now obviously been fixed and we’re back in action. Thanks again to those who let us know.

I’m very pleased to say that the response has been great to my request for new contributors to this site and over the past few days I’ve given author credentials to nine new authors. They represent a great range in experience and geography. A couple are seasoned VoIP/communication security professionals who have been around VOIPSA circles for a while and in a couple of cases have written books on the topic. (Some I’ve written about here or interviewed on Blue Box.) Others have been involved in security or VoIP but haven’t really had a profile in “VoIP security”, per se. And there are a couple who are brand new to the field but have some great passion to contribute.

I’m also pleased that we’ve added a couple of Europeans so that Martyn Davies is no longer holding down the fort as the only non-US regular contributor. We’ve also added our first contributor from India (or for that matter anywhere in Asia). While the vast majority of VoIP security issues have no relation to geography, there are of course laws and regulations that come up in different regions, as well as regional news items, and so it is nice to have a wider geographical distribution.

Thanks again to all who responded (and we’re still open to others) and we look forward to the additional posts they may bring over time.

Our whole goal with this site is to create conversations around VoIP / communications / UC / SIP security regarding what the issues are, what the “real” dangers are (as opposed to those sometimes hyped in the mainstream media), what the solutions are, etc. so that in the end we will all have safer and more secure communication systems.

Thanks to all of you – both writing and reading – for joining in that conversation.

If you found this post interesting or helpful, please consider either subscribing via RSS or following VOIPSA on Twitter.

Yes, indeed, the VoIP Security Alliance has joined the Twittersphere with:

http://twitter.com/voipsa

Feel free to follow us there if you are a Twitter user. The primary reason we are on Twitter is so that Twitter users can follow whatever blog posts we post here on the Voice of VOIPSA blog. We’ve noticed over time on other sites (and in our own actions) that some folks prefer to be notified of new blog posts via Twitter versus a RSS feed. So now you have that choice. Subscribe via RSS or via Twitter. We’ll respond to tweets as well, of course, but our primary goal is to provide another way to consume VOIPSA content.

If you are on Twitter, please do feel free to follow us. Thanks.

Are you interesting in writing about VoIP security? In providing updates on security news? Product reviews? Threat analyses? Notes about recent security advisories?

Would you like your writing to appear on this blog?

As you have probably noticed, the frequency of our posting here in recent months has dropped a bit. It’s definitely not for lack of content… anyone subscribing to a Google Alert on “voip security” or subscribing to the VOIPSEC mailing list will know that there are definitely ongoing VoIP security issues. But we collectively haven’t been writing all that often about those issues here on this blog. Many reasons… but mostly that those of us who have been writing for the three years since we started this blog have just been finding ourselves insanely busy and not able to make the time to write here frequently. A couple of folks have moved into roles where they no longer work directly with VoIP security. Others have started their own blogs or just gone on to other things.

So we are looking to recharge the “Voice of VOIPSA” writing corps a bit. Our goal all along has been to make this site a portal for news and analysis about “VoIP security” in whatever form that may take. We are looking for people who might be willing to write short notes about news stories related to security of VoIP, Unified Communications, etc. We are also looking for people interested in writing longer pieces like some of the deep analyses we have posted here in the past.

VOIPSA’s overall mission is to raise the level of discussion about communication security issues in the IP space – and we’re looking for anyone who would like to help us in doing that through this blog.

The only major requirement we have for writers here is that any pieces must be vendor-neutral, i.e. we are not looking for people to write here about how their company’s product will solve all your security woes. We’re not a marketing site for either VoIP or security vendors. However, we do welcome posts from people at those companies that talk about the general state of the industry. We also welcome posts from folks who may not be at any company in the space but are just passionately interested in the topic.

If you are interested in writing for Voice of VOIPSA, please send me an email expressing your interest and providing some background about your connection to VoIP security. If you write at an existing weblog, even on a completely different topic, it would be helpful if you sent along that link as well.

Thanks for continuing to follow this site and after three years of blogging, we’re looking forward to continuing to provide you information and analysis about VoIP/communication security for the next three years… and beyond!

Technorati Tags:
voipsa, voip, voip security, voipsecurity, security, blogging

As some of you may have noticed, our servers were offline for the past 24 hours due to unforeseen circumstances.  It seems the recent global economic turmoil has not left VOIPSA unscathed.  Turns out our hosting provider was delinquent on paying their bills to their upstream data center provider.   Supposedly, the hosting provider’s management is no where to be found and did not respond to repeated billing inquiries,  leaving the upstream data center no choice but to unplug all of the hosting provider’s customers.

Apologies for the inconvenience and we’re working on moving to a more permanant and solvent hosting provider in the near future!

This site is now updated to use WordPress 2.7, the newest version of WordPress.  Everything seemed to go okay but if you do see any issues with the site please let me know via a comment here or via email.

For those of you who may be used to reading this blog through the “Security Bloggers Network” set up originally by Alan Shimel, you need to be aware that the “SBN” is going through a transition. As Alan details on his blog, Google is in the process of shutting down the “Network” feature of Feedburner and as a result the page and feed for the SBN will be going away.

Alan is working on a new solution but in the meantime you may want to grab the OPML file for the Security Bloggers Network (you should then be able to import this into most feed readers). There are a lot of great security blogs out there.

Stay tuned for more information – once Alan has another solution in place I’ll post an update.

We’ve upgraded this site to WordPress 2.6.2… there shouldn’t be any issues with the site, but please let us know if any pages act strange. Thanks, Dan

Not wanting to get into any of the problems we had previously, I’ve gone and upgraded this site to be running the newly-released WordPress 2.6. If you see anything strange going on with the site, please do let me know. Thanks.

This blog site was hacked. Cracked. Whatever you want to say. We appear to have been hit by spammers / black hat SEO types. It turns out that we are not alone. So let’s talk about what happened and why.

First, though, if you use WordPress on your blog site and have NOT yet upgraded to WordPress, 2.5, STOP reading this article and go upgrade! We’ll still be here when you’re done.

WHAT HAPPENED?

In the last week of March, an attacker (or attackers) compromised our site and in a particularly insidious attack, added some text that was invisible to the viewer of our web pages, but was in the source of the file and therefore was seen by spiders from Google, Technorati and other search sites. If you looked at the pages of our site, you would not have a clue that they were the host to all sorts of spam links. However, if you looked at the source code, you would see something like this at the bottom of any given page (displayed as a screen capture so as not to give the spammers any more links):

Why was it “invisible”? Simple, the attackers simply added a ’style=”none”‘ attribute to an HTML tag, in this case the good old <U> tag (underline):

That was it. Add a “style” attribute and in this era of stylesheet-aware browsers (which generally is a good thing), the text was invisible to the reader of the blogs.

WHY DID IT HAPPEN?

Fairly simple answer. We were still running a comparatively ancient version of WordPress… version 2.1.2. We had not upgraded to one of the more recent 2.3.x builds (although there are indications there are security issues with 2.3.3 as well (also here)). Yes, this is particularly embarrassing for us because we are a security organization but in a classic case of the “cobbler’s shoes” we were not staying up-to-date with the software here. VOIPSA is a volunteer, nonprofit organization and while that is not an excuse, that may explain it. There are a couple of us who do the system administration for VOIPSA’s site and we had discussed upgrading several times but given that we’d had some problems with earlier upgrades we wanted to have a block of time to do the upgrade – and we didn’t make that time. And as a result we were hit.

HOW DID IT HAPPEN?

It appears that the attackers used some type of PHP upload vulnerability to upload files to our site. We noticed a number of very small files that were uploaded which perhaps were their test files. We don’t (yet, anyway) know precisely what vulnerability the attackers exploited, but in looking at the changelogs for various versions of WordPress since the time of 2.1.2 it is very clear that several such vulnerabilities have been fixed in newer versions.

What the attacker(s) ultimately did, though, was to modify the “footer.php” file to include this “invisible” text above. Now where they did was a key factor. The first thing we did when we saw this was to check the ‘footer.php’ in the WordPress theme we are using for this blog. We use a very slightly modified version of the widely used Kubrick theme, primarily to bring in the graphic on the top, and it resides in its own directory. However, the modification was not there. We did some further searching and found the modifications in the “footer.php” file located in the “default” theme.

This puzzled us for a bit because we don’t use the “default” theme, but it would appear that the “wp_footer()” function must also call the footer.php file in the default directory as well as our own. I haven’t honestly crawled through the PHP code to figure this out, but by what we saw it would appear that this is the case.

Given that, the attacker’s task was fairly easy:

1. Find an older WordPress system with exploitable vulnerabilities.

2. Upload the modified file to “wp-content/themes/default/footer.php”

Ta da… invisible links are now there for sleazy SEO purposes.

HOW DID WE NOTICE IT?

Two ways. First, something the attackers did broke the web GUI editor. We still don’t know exactly what caused this (without going through all the WP code), but I knew something was wrong with the site when one of the other contributers contacted me to say he could not post a blog entry. You could go in and write the entry, but when you hit the “Publish” button you wound up with a completely blank screen. Something was causing the post.php file to fail. I didn’t notice this myself because I do all my writing offline (using MarsEdit) and that worked perfectly fine through the API. In looking into the problem with the editor, though, we saw the spammy links on the bottom of the posts and investigated further.

Second, the attackers seemed to get a bit cocky. They directly modified one of the blog entries. More than just the footer, they went in and modified one of the entries to include a couple of visible links. About the same time we were digging into the footer issue, one of the other contributors contacted us asking what was going on with all the spam links. Now it’s not clear that this was an exploit of the same vulnerability or of a different one, but it certainly clued us in to their being a problem.

WHAT HAVE WE DONE TO FIX IT?

First, we’ve upgraded to WordPress 2.5.

Second, we’ve gone through all the code in our themes (both the one we use and the default theme) ensuring there is no more bogus code (the upgrade seemed to take care of it all). We’ve also gone through our database to make sure there is no bogus text there.

Third, we’ve installed the “WordPress Automatic Upgrade” plugin which makes the WordPress upgrade process incredibly painless. Now that we have that plugin installed, it will be very easily to stay up-to-date with the very latest versions of WordPress. Note that it doesn’t automatically upgrade the site when a new version of WordPress comes out (we’re way too paranoid to allow that!) but it automates the tasks involved with an upgrade: backing up the themes and database, downloading the current version, de-activiating an re-activating plugins, etc. All the manual steps are now quickly done by the plugin.

NOTE TO WORDPRESS TEAM: Have you considered building this plugin directly into WordPress?

Fourth, we are giving a second look to hosted platforms (like WordPress.com) purely so that someone else can be responsible for system administration and upgrades – leaving us to just write. Being control freaks with a certain level of paranoia (as most security people are), we have resisted this in the past – and may still – but will be taking another look.

We are also suitably chastised (and angry) at having the site attacked so we’ll be having a renewed sense of vigilance with ensuring that this does not happen again.

FINAL THOUGHTS

The sad reality is that the spammers out there will always be looking for new and creative ways to game the system and do their dirty work. The reality is that they make money doing this for their clients and have every incentive to continue. The challenge we all have as operators of sites is to continue to try to stay ahead of the spammers – and to stay up-to-date with the software!

I would also just end with a word of thanks to all the WordPress developers who continue to fix security issues as they are found and who have made it such a great platform for blogging. I’d also thank the folks at Technorati who have made use of their massive database of blogs to notify people with vulnerable WordPress blogs (I received several email notices this morning) and who have stopped indexing afflicted blogs (denying the spammers a few of their links.)

Finally, I would encourage folks to read this article and it’s update over on the blog site Deep Jive Interests as they get into more about these attacks that have been going on across the blogosphere.

P.S. And if you haven’t upgraded to WordPress 2.5 yet, why not?

Technorati Tags:
wordpress, security, wordpress security, wordpress hacks, voipsa