As I mentioned last week, I’ll be participating in an Ingate-sponsored webinar on “Secure SIP Trunking” in about 2 hours at 2:00pm Eastern US time. If you’d like to learn about VoIP security in general (as well as obviously Ingate’s products), you can participate in the free webinar. Sign up here.
Technorati Tags:
sip, security, sip security, voip, ingate, voip security, voipsa, sip trunking
This blog site was hacked. Cracked. Whatever you want to say. We appear to have been hit by spammers / black hat SEO types. It turns out that we are not alone. So let’s talk about what happened and why.
First, though, if you use WordPress on your blog site and have NOT yet upgraded to WordPress, 2.5, STOP reading this article and go upgrade! We’ll still be here when you’re done.
WHAT HAPPENED?
In the last week of March, an attacker (or attackers) compromised our site and in a particularly insidious attack, added some text that was invisible to the viewer of our web pages, but was in the source of the file and therefore was seen by spiders from Google, Technorati and other search sites. If you looked at the pages of our site, you would not have a clue that they were the host to all sorts of spam links. However, if you looked at the source code, you would see something like this at the bottom of any given page (displayed as a screen capture so as not to give the spammers any more links):
Why was it “invisible”? Simple, the attackers simply added a ‘style=”none”‘ attribute to an HTML tag, in this case the good old <U> tag (underline):
That was it. Add a “style” attribute and in this era of stylesheet-aware browsers (which generally is a good thing), the text was invisible to the reader of the blogs.
WHY DID IT HAPPEN?
Fairly simple answer. We were still running a comparatively ancient version of WordPress… version 2.1.2. We had not upgraded to one of the more recent 2.3.x builds (although there are indications there are security issues with 2.3.3 as well (also here)). Yes, this is particularly embarrassing for us because we are a security organization but in a classic case of the “cobbler’s shoes” we were not staying up-to-date with the software here. VOIPSA is a volunteer, nonprofit organization and while that is not an excuse, that may explain it. There are a couple of us who do the system administration for VOIPSA’s site and we had discussed upgrading several times but given that we’d had some problems with earlier upgrades we wanted to have a block of time to do the upgrade – and we didn’t make that time. And as a result we were hit.
HOW DID IT HAPPEN?
It appears that the attackers used some type of PHP upload vulnerability to upload files to our site. We noticed a number of very small files that were uploaded which perhaps were their test files. We don’t (yet, anyway) know precisely what vulnerability the attackers exploited, but in looking at the changelogs for various versions of WordPress since the time of 2.1.2 it is very clear that several such vulnerabilities have been fixed in newer versions.
What the attacker(s) ultimately did, though, was to modify the “footer.php” file to include this “invisible” text above. Now where they did was a key factor. The first thing we did when we saw this was to check the ‘footer.php’ in the WordPress theme we are using for this blog. We use a very slightly modified version of the widely used Kubrick theme, primarily to bring in the graphic on the top, and it resides in its own directory. However, the modification was not there. We did some further searching and found the modifications in the “footer.php” file located in the “default” theme.
This puzzled us for a bit because we don’t use the “default” theme, but it would appear that the “wp_footer()” function must also call the footer.php file in the default directory as well as our own. I haven’t honestly crawled through the PHP code to figure this out, but by what we saw it would appear that this is the case.
Given that, the attacker’s task was fairly easy:
Ta da… invisible links are now there for sleazy SEO purposes.
HOW DID WE NOTICE IT?
Two ways. First, something the attackers did broke the web GUI editor. We still don’t know exactly what caused this (without going through all the WP code), but I knew something was wrong with the site when one of the other contributers contacted me to say he could not post a blog entry. You could go in and write the entry, but when you hit the “Publish” button you wound up with a completely blank screen. Something was causing the post.php file to fail. I didn’t notice this myself because I do all my writing offline (using MarsEdit) and that worked perfectly fine through the API. In looking into the problem with the editor, though, we saw the spammy links on the bottom of the posts and investigated further.
Second, the attackers seemed to get a bit cocky. They directly modified one of the blog entries. More than just the footer, they went in and modified one of the entries to include a couple of visible links. About the same time we were digging into the footer issue, one of the other contributors contacted us asking what was going on with all the spam links. Now it’s not clear that this was an exploit of the same vulnerability or of a different one, but it certainly clued us in to their being a problem.
WHAT HAVE WE DONE TO FIX IT?
First, we’ve upgraded to WordPress 2.5.
Second, we’ve gone through all the code in our themes (both the one we use and the default theme) ensuring there is no more bogus code (the upgrade seemed to take care of it all). We’ve also gone through our database to make sure there is no bogus text there.
Third, we’ve installed the “WordPress Automatic Upgrade” plugin which makes the WordPress upgrade process incredibly painless. Now that we have that plugin installed, it will be very easily to stay up-to-date with the very latest versions of WordPress. Note that it doesn’t automatically upgrade the site when a new version of WordPress comes out (we’re way too paranoid to allow that!) but it automates the tasks involved with an upgrade: backing up the themes and database, downloading the current version, de-activiating an re-activating plugins, etc. All the manual steps are now quickly done by the plugin.
NOTE TO WORDPRESS TEAM: Have you considered building this plugin directly into WordPress?
Fourth, we are giving a second look to hosted platforms (like WordPress.com) purely so that someone else can be responsible for system administration and upgrades – leaving us to just write. Being control freaks with a certain level of paranoia (as most security people are), we have resisted this in the past – and may still – but will be taking another look.
We are also suitably chastised (and angry) at having the site attacked so we’ll be having a renewed sense of vigilance with ensuring that this does not happen again.
FINAL THOUGHTS
The sad reality is that the spammers out there will always be looking for new and creative ways to game the system and do their dirty work. The reality is that they make money doing this for their clients and have every incentive to continue. The challenge we all have as operators of sites is to continue to try to stay ahead of the spammers – and to stay up-to-date with the software!
I would also just end with a word of thanks to all the WordPress developers who continue to fix security issues as they are found and who have made it such a great platform for blogging. I’d also thank the folks at Technorati who have made use of their massive database of blogs to notify people with vulnerable WordPress blogs (I received several email notices this morning) and who have stopped indexing afflicted blogs (denying the spammers a few of their links.)
Finally, I would encourage folks to read this article and it’s update over on the blog site Deep Jive Interests as they get into more about these attacks that have been going on across the blogosphere.
P.S. And if you haven’t upgraded to WordPress 2.5 yet, why not?
Technorati Tags:
wordpress, security, wordpress security, wordpress hacks, voipsa
On April 1st VuNet reported that hackers had taken down the International Space Station’s email capabilities.
So, this was a good April Fool’s joke, right?
Three astronauts onboard the Space Station reported last night that email was no longer working.
Hackers are thought to have planted a Trojan in the computer systems at Houston and used the infection to ride the satellite uplink to the Space Station.
What is especially troubling is the email system’s reliance upon older Microsoft operating systems that are no longer supported by Microsoft.
“I am sorry but there is nothing we can do. It is past its deadline, said Professor Brian Offin, Microsoft’s head of obsolete operating systems.
Again, a good April Fool’s joke, right?
However, this false article brings to light the fact that as newer technologies replace legacy systems, we must bear in mind that the new technology changes will, over time, themselves become legacy systems and subject to the same outdated, unsupported and insecurities that plagued the very legacy systems they replaced.
So what’s this have to do with VoIP and the International Space Station? Well, details are thin, but way back in 2000 VoIP Group Inc. was awarded a contract to provide a VoIP replacement for the ISS to “bring about significant cost reductions as it supplements and then replaces an existing legacy system.”
Initially deployed at NASA’s Marshall Space Flight Center in Huntsville, Alabama, and later at other International Space Station operations centers, the solution will consist of VoIP Group’s gateways connected to the Internet and to Raytheon voice switches and CUseeMe conference servers to support voice conferencing. The system is designed to link together researchers, NASA operations personnel, and potentially ISS crew, to support collaboration during Space Station experiment planning and operations. Because users can access the system using a standard Internet browser on an inexpensive multimedia PC, they can be located at NASA centers, universities, and companies throughout the world, and still connect in real-time, 24 x 7.
I hope that the sharp folks at NASA and VoIPgroup are taking the proactive steps to avoid security problems with critical communications with the ISS.
I do have to hand it to the VoIPshield Systems folks… they really did go all out for their product launch. As I noted yesterday, they released a slew of vulnerability notices… but I didn’t notice that they also released a YouTube video “dramatizing” a potential DoS attack by someone connecting to a lobby phone. It was a Network World article that pointed me to it:
I have to say that this is the first time that I can personally remember a “VoIP security video” being uploaded to YouTube by a company doing a product launch (although Peter Cox did upload one as he was launching his consultancy). It’s also the first “dramatization” I recall seeing. (Peter’s and others (including mine) have been more documentary/interview style.)
So kudos to VoIPshield for doing something a little bit different. Nice to see.
I’m also a huge fan of telling stories as a way to talk about issues in general, so it’s good to see.
As to the video itself, I had the following comments:
While the video is more on the alarmist side of the security continuum than I am (but, gee, what does VoIPshield sell?), it’s nice to see someone doing something a bit offbeat and different in trying to talk about VoIP security issues.
I look forward to seeing VoIPshield’s next video…
Technorati Tags:
voip, voip security, security, voipshield, videos
How can you make SIP trunking secure? Is there such a thing as “secure SIP trunking”? Can SIP trunks and VoIP actually be more secure than the PSTN?
All those questions and more will be the subject of a webinar next week sponsored by Ingate Systems (and announced today) in which I will be a participant called “Secure SIP Trunking: What You Need to Know“. The webinar will cover:
Now, obviously this webinar is sponsored by Ingate so the solutions offered will involve their products. My role will be to talk about VoIP security in general and issues around securing SIP trunks. It should be an interesting session and you can easily register if you would like to attend. There is no charge.
The webinar will be on Thursday, April 10th, at 2:00pm US Eastern time, 11:00am US Pacific time.
NOTE: VOIPSA does not directly endorse, recommend, or promote products from any vendors. Our mission is to raise the level of discussion around VoIP security issues and so we are glad to participate in any relevant educational efforts such as webinars, conferences or other events. We are participating in this and other events sponsored by Ingate simply because they asked us and the events seemed in line with our overall mission. If you would like VOIPSA participation in an event you are sponsoring, please contact a VOIPSA Board Member about the possibility.
Technorati Tags:
voip security, voip, sip, sip security, sip trunking, sip trunking security, ingate, voipsa
So today VoIPshield Laboratories announced the discovery of over 100 security vulnerabilities in systems from Avaya, Cisco and Nortel and, somewhat predictably, this has already resulted in coverage from the Wall Street Journal BizTech blog and InfoWorld’s Security Watch blog. I will expect to see more coverage in the days ahead as it works its way out into the mainstream media. The news release is a good one and includes great quotes from Gartner’s Lawrence Orans and our (VOIPSA) own Jonathan Zar.
VoIPshield has disclosed all the vulnerabilities to the vendors and has made 44 of the vulnerabilities available at www.voipshield.com/research (That is the number I currently see on the page.)
I should note that VoIPshield Systems, under whom VoIPshield Laboratories falls, is a member of the VOIPSA Technical Board of Advisors. I also have met the VoIPshield folks several times and Jonathan and I interviewed CTO Bogdan Materna eons ago back on Blue Box podcast #12. We’ve been on panels together and I have a high degree of respect for what they are doing and how they are doing it.
This familiarity, though, does not prevent me from feeling a bit uneasy about two aspects of this announcement today. First, if you look down the list of vulnerabilities in almost all cases (41 of 44) the vendor response state is “Attempting to address the issue“.
In other words, these are current, open vulnerabilities. No patches. No fixes. (Outside of the stated recommendation to follow network security best practices and potentially to purchase a VoIP security product such as the one VoIPshield makes.)
Now in many cases the vulnerability announcements are sufficiently vague that an attacker is not going to be able to do a whole lot with them. However, in other cases, there’s enough information there to point the way for an attacker. For instance, this one for Cisco for “UCM Multiple Hardcoded Passwords” indicates:
Description
By knowing and using the hardcoded account names and passwords (a total of three have been identified) on the UCM platform, an attacker can connect to the system and issue database commands which can result in code execution, denial of service, license exhaustion or theft, etc.
So now we know there are three account names (at least) with default passwords that can be used to administer a Cisco UCM system. How long will it be now before someone sitting there with a brute-force password script will figure out those names and post them to one of the various default password lists out there?
Now, this particular vulnerability announcement does state:
Cisco acknowledges the presence of these hardcoded passwords and is working to have the values set to an administrator definined setting during installation.
This will undoubtedly involve a new release of the software (since it refers to the installation process). That will take some time, obviously, and in the meantime any Cisco Unified Communications Manager installations out there are potentially vulnerable to abuse through these hardcoded usernames and passwords.
I understand that VoIPshield did contact these vendors and at least per the WSJ article gave them at least 30 days notice. I also realize that vendors may not always be able to create quick solutions and also may not assign the same priority to issues (or may in fact dispute/dismiss the issue). Having been on the vendor side, I well understand the dynamics of working with security research firms. I know there can be challenges on both sides. Still, I personally would have been a lot more comfortable with seeing this information out there if they had waited a bit until more than just 3 of the 44 listed vulnerabilities have vendor patches available.
Which brings me to my second concern. The vulnerability notices posted do not include any “mitigating circumstances”. They state the description of the problem and offer the recommendation to use network security best practices and VoIP security products such as those sold by VoIPshield, but they do not provide a sense of how to evaluate the risk involved. For instance, with the hardcoded passwords, I am assuming the attacker needs to be on the internal network, but is that correct to assume? With the DFR Cancel Backup Command Injection vulnerability, does the attacker need to be on the internal network? Or could they be on the public Internet? (if systems traversed the Internet)
Now perhaps VoIPshield is waiting to provide this type of information until there is a fix out there. This is also their first time issuing public vulnerability notices in this form. Perhaps with feedback such as this they will provide that added information. But without that kind of information, it’s not clear to me that I have enough information to understand the potential risk to my systems. (And perhaps we’ll have to have them on a podcast to talk about all of this.)
Regardless of these two concerns, the fact remains that VoIPshield Laboratories has entered the space as a new research entity and has already brought out a wealth of research. Per their news release, this is just the first step and more information will be coming soon. That all is good to hear as having such research groups focusing on the VoIP security space is a definite good thing. We need more research in the field and so it’s great to see VoIPshield entering the space publicly. (They’ve obviously been doing this research privately for their products for some time.)
And, if you are the administrator of a system from Avaya, Cisco or Nortel, I would strongly encourage you to review the vulnerabilities and try to understand which of them may or may not affect your installations.
Technorati Tags:
voip, voip security, security, cisco, avaya, nortel, voipshield
FYI, this site is now running WordPress 2.5 and everything seems to be fine. If you experience any issues commenting on posts or see anything strange about how the site appears, please do let us know.
We know return you to your regularly-scheduled VoIP security news…
P.S. I should note that we’ve now installed the “WordPress Automatic Upgrades” plugin and it worked very well to upgrade this site to WP 2.5. We’re not adventurous enough to let it just automatically upgrade the site by itself, but we’ll certainly use it to manually upgrade the site in the future.
Just FYI, we’re going to be upgrading this site to WordPress 2.5 sometime in the next couple of days. There should be no noticeable impact to the site but if you land here sometime and the site looks a bit strange, it may be because we’re in the middle of the upgrade. (We do not expect to have any issues moving the theme we use to WP 2.5, but we don’t know.)
It’s refreshing to see a vendor in the IP phone space respond to reported security problems with their products. During the GNUcitizen Router Hacking Challenge several issues were reported with the Snom 320. The vulnerabilities posted were also picked up by Tom Keating’s blog. Gnucitizen posted a webpage detailing the vulnerabilities as well, and the vendor response has been very good, with the following actions taken by Snom (note: typos left in):
So, this is a good start, but I do have a few humble suggestions for Snom:
Overall, this is encouraging to see a VoIP phone vendor stepping up and taking ownership of product vulnerabilities – Kudos to Snom!