Author Archives: J. Oquendo

About J. Oquendo

Security practitioner with a broad range of experience and expertise in networking, security, systems administration/design/engineering, VOIP, application and systems exploitation. Working at a technology (cloud) company that offers various managed security based services. Responsible for Incident Handling, Penetration Testing, Security and Information Risk Assessments, Risks Analysis/Management, Audits and Compliance testing. I no longer publicly disclose the companies I work for – but feel free to ask.

VoIP Attackers – Sometimes They Come Back

In my previous post I briefly touched on information that I had been collecting for over 19 months via what began as VoIP Intrusion Detection system I had whipped up from scratch. The framework for Arkeos began almost three years ago while working here at an ITSP slash managed service provider. One of our services is a “managed VoIP service” similar to Packet8 or Vonage for small to mid-sized business. I lost count of how many PBX’s we maintain nowadays but I will comfortably float this number at around 60’ish. Altogether I have about 1,600 clients mainly small to mid-sized businesses with a couple of Fortune 500’s in the mix. My company also provides SIP trunks and we have about 30-35 VoIP companies who use us as their backbone for their VoIP services. Because of my position in my company (multiple roles), I have great visibility into many different VoIP hardware and software vendors in fact, I can’t think of a VoIP related vendor I haven’t dealt with in the past 5 years. Avaya, Nortel, Mitel, Cisco, Acme Packet, Audiocodes, Allworx, PBXNSIP, Asterisk and the list goes on. Because my “main cup of tea” is security, I began meshing security with VoIP.

Arkeos does a few things: 1) Collects information on bruteforcers 2) Collects information on attackers 3) Records the calls an attacker is trying to send and 4) Mimics a phone so that an attacker thinks they are placing a valid call. When I first scripted it, I had to customize it in accordance to the device it was being deployed on, these were mainly Asterisk based PBX’s. Because I was using different versions of Asterisk, I had to tweak the script almost daily as the columns in Asterisk’s message file was different. For example:

2010-09-25 21:02:48 NOTICE[14155] chan_sip.c: Registration from '"3068" ' failed for '72.55.179.180' - Wrong password
2010-09-25 21:02:50 NOTICE[14155] chan_sip.c: Registration from '"3068"' failed for '72.55.179.180' - Wrong password

These are two different entries to parse with awk. To obtain the first entry, I would use ‘awk {print $10}’ whereas the other would be awk ‘{print $9}’. I initially thought of going with Python, then Perl, but settled on good old fashioned shell scripting. This ensured me that no matter what system I deployed it to, I would be able to use readily available tools on the system without introducing another application. Arkeos in all honesty is a nightmare to look at if you’re a coder, yet completely functional for what it does.

After tweaking it to my needs, I began to see many patterns die and many emerge. Years back, I saw the “fire and forget” method of attacks where you had a bruteforcer compromise an account, then that same bruteforcer fire of as many calls as they could possibly send. Nowadays I see “bruteforce” drive-bys and an attacker come from left field. Without giving up information to potential attackers who may read this, let’s assume I have an account of 5000 with a password of 5000, I’ll see say 10,000 attempts from an address of 10.10.10.10 trying to guess the password, with an attacker from 20.20.20.20 registering to use the account. From more or less November of 2009 through now, I have yet to see a bruteforcer gain an account AND log in at the same time. On the contrary, I don’t see bruteforcers’ addresses on my honeypots anymore.

mysql> use arkeos;
Database changed
mysql> select * from attackers where attacker_address='92.81.245.67';
ZZZZ, 92.81.245.67, Zoiper rev.5324, XXXX, 2010-09-11, 03:16:28
1 row in set (0.00 sec)

mysql> select * from bruteforcers where attacker='92.81.245.67';
Empty set (0.01 sec)

mysql> select * from bruteforcers where start_date='2010-09-11';
Empty set (0.00 sec)

mysql>

NOTE: I modified the output to fit it to page, the exact format of Arkeos and its tables can be seen at: http://infiltrated.net/arkeos-w-mysql.txt and the audio for this call can be heard at http://infiltrated.net/mgz/92-81-245-67-audio.wav)

So why is this information not only important to engineers (be they VoIP, Security, Network or Systems engineers) but also to perhaps law enforcement, the answer is a bit complex so I will try my best to explain my point of view, theory and concept. Imagine that right now I saw a bruteforce attack from say Korea with the account actually being used from say Romania. Romania hoses my machine and costs me say 100,000.00. For the time it would take for an investigation to even take place, get subpoenas in order, go around the world, get those companies to co-operate, at the end of the day, the investigation would likely yield NOTHING. I say this because of the ease of readily available networks and hosts that are likely compromised and used for bruteforcing and attacking. However, this is SOLELY for the bruteforcers. Did I mention the actual attacker is coming from elsewhere?

An attacker is, who I categorize in my writing, someone who has actually tried to place a call through my honeypot. Heck, bruteforcers I can easily block, prior to my honeypot I had created my own Asterisk based IPS capable of blocking bruteforcers in real-time. Attackers I want to believe are a bit more static in the sense that I believe whoever is behind 92.81.245.67 is the culprit who needs to be either arrested or rubber-hosed (depending on your view of justice). So what is relevant for an investigator outside of the attacking IP? For starters, Zoiper is a softphone (for those who don’t already know this) which means this attacker is likely using either a desktop or a laptop. Further, because I’m also tracking the number(s) being called, I gained the following insight:

awk '/407272/{print $1}' /usr/share/arkeos/calls/92.81.245.67/*
0040727246641
01140727246641
00140727246641
901140727246641
90040727246641
90040727246641
90040727246641

What have we here? Intervention is what. Arkeos mimics a telephone so unless an attacker was on the opposite end of the number dialed, how would they know to try 011, 001, 9011, 900 and so on and so forth. Hence the title “Sometimes they come back.” I have seen quite a few attackers coming back week after week from the same addresses sometimes, using the same accounts, trying to call the same number. So where would I look if I were an investigator? Directly at whomever owns that number they’re trying to call, that’s where. I theorize that if one tracked the owner of that number, searched the computers of who owned that number for Zoiper, I’m willing to bet they’re one in the same. Coincidental? Highly unlikely. An IP address, a phone number AND a softphone client ALL coincidentally belonging to one person is more than coincidental.

Anyhow, I have created an online submission form for others to submit attacker data and am trying to figure out a way I can streamline Arkeos to be a “plug and play” script for anyone willing to use it. I will eventually create a database which others who collaborate can update and pull from but I don’t want to run into a thousand emails asking me to “customize” Arkeos. This is the only reason I haven’t made it public. Remember, I made this from scratch and it was tailored to my machines and since all machines differ, I do know that I will receive those emails. To which I have little time to respond to. I try to answer as many emails as I can, but the fact is, I’m all over the place sometimes (literally.)

VoIP Abuse Project

Brief History

Throughout the course of three years I have been studying the Voice Over IP attack canvas. As a security engineer, I have had the privilege of seeing an enormous amount of attacks while working at a Managed Services Provider. Because I have multiple roles in the company I am working at, I get the best of multiple worlds. Sometimes I’m a VoIP engineer, sometimes a pentester, sometimes systems engineer, other times it depends highly on my mood. Overall I’m a Senior Coffee Tasting Engineer working at a company that gives me great flexibility to do what I want. This allowed me to create my own Voice Over IP Honeypot I called Arkeos for data analysis.

My original security application began as a “blacklisting” slash “VoIP Intrusion Prevention System” program using Asterisk. The goal was simple, keep an eye on live servers and when an attacker made a move, initiate a mechanism to automatically block the attacker, send the engineers a report, place the offender in a list to be distributed to all other PBX’s in my company so they too can block them. Initially my VoIP IPS worked just as programmed however, because I have clients all over the world, it became difficult to outright block entire CIDR ranges. I quickly whitelisted many addresses, re-hashed the IPS which worked fine for years.

The Arkeos idea came to mind after browsing around to see that there really wasn’t anyone to correspond with in this matter. People looking for and sharing similar data were few as opposed to normal networking slash abuse. I decided to build my honeypot based on what I was seeing from monitoring logs. Anyone who has known me for years can tell you that I kept tons of terminals opened with good old faithful tail -f running alongside whatever logging applications I have in house (Splunk, etc). As months passed by, I began archiving data from attacks spread across 20-30 Internet facing PBX’s. My current company sells VoIP trunks, managed VoIP services, T1-OCx connections, managed security services, video conferencing, digital signage, etc.; if it’s out there, then we likely do it. This gives me a high degree of visibility for not only VoIP attacks, but other attacks as well.

Arkeos is a program primarily used on Unix based servers running the Asterisk Open Source PBX. The application mimics a valid extension in which an attacker, after bruteforcing an account, attempts to place calls. What the attacker doesn’t know is that their calls go nowhere. The sole purpose of allowing them to perceive they have an account is to track them. Where they come from, what accounts they use, what VoIP account scanners they use, what numbers they try to dial. Because of the flexibility of Asterisk and Unix as a whole, the application does a lot more than stated however, I will not disclose too much otherwise any attackers reading this might catch on. Then I won’t have anyone to play with.

As time progressed customers who purchased SIP trunks from us were getting “owned” from time to time and to date, I have been called to handle incident response and forensics on over 75 VoIP servers ranging from Asterisk, Avaya, pbxnsip, CME and a mixture of everything in between. Usually the outcome has been the same: weak accounts and weak passwords.

2007 – 2008’s Attackers trends

For the better part of 2007 and 2008, attackers had a common modus operandi: scan a host for an open account, register a device (soft phone, another SIP trunk) and fire away thousands of calls. I have personally seen a company spend over a quarter of a million US dollars because of these attacks. Before I dive more into stats a bit, I will explain a bit about the financial loss for those who don’t think that it’s a big deal. I currently have a client who purchases a heavy amount of SIP trunks from my company, they in turn have a “Vonage” like business. Their customers buy an ATA, number, connect and place calls from Latin America. My client one day sends me an e-mail asking me if I would be willing to take a look at his network as his upstream was sending them abuse messages. Game on.

My client has approximately 20 servers in the continental US and 20 in Latin America. They were using a mixture of Asterisk and other equipment. What was happening to their servers was two-fold, attackers were leveraging VoIP accounts and placing thousands of calls. During the times when attacks took place, their bandwidth was so saturated that normal clients’ calls experienced horrible quality and drops. This led to loss of clients, enormous phone bills to places like Sierra Leone, Romania and the list goes on. To make a long story short, it took me approximately six weeks to clean up their shop as I could NOT take any of their servers offline. I essentially had to analyze what occurred, cut the bleeding, implement a fix and implement post remediation. Ultimately I programmed a distributed “cleaning” application to clean up accounts, create appropriate firewall rules, clean up backdoored binaries, implement an SIEM. This was another cost the client had to contend with. When I say over a quarter of a million dollars I am being very conservative. That was only the first instance of what occurred in their network, the second was, a few of their servers had become bots – but that is another story. Moving back into the statistics and trends of attackers I’ve seen, attackers are becoming smarter and the attacks seem more focused and targeted.

2009 – 2010’s Attackers trends

On a daily basis I see approximately six or seven attacks on each one of my Internet facing PBX’s, this estimate is based on solid numbers. The total amount of attacks divided by the amount of servers I have. While it may not seem like a big deal but the fact of the matter is, if one attack cost a company a quarter of a million in damages, the last thing I need to see is an attack on my company. Management’s first response would be “why didn’t you stop it.” The reality is, my company too was the victim of tollfraud via legacy VoIP accounts. We experienced 96 billable hours from our carrier to calls made to numbers in Sierra Leone. I know firsthand that these attackers are not “script kiddies” but focused fraudsters. Altogether during a highly volatile week, I can see approximately 150 – 200 attacks per day.

Previously (between 2007 – 2008) when I monitored the attacks, the attack phase was: “get an account, fire and forget,” whereas now, there seems to be less automation and more intervention. This is evident in the way attackers are placing calls. Here is an entry from today in my Arkeos database of an attacker coming from an address in Germany. Take note that I have sanitized data to keep the identity of my honeypot at bay:


# more peer-verbose-Mon-21-Sep-2009-11\:59\:34-AM-EDT.txt
 -- General --
           Name: SIP/300-b6b241c0
           Type: SIP
       UniqueID: 1253548774.7688
      Caller ID: 1xxxxxxxxxx
 Caller ID Name: xxxxxxxxxx
    DNID Digits: 16085541772
          State: Ring (4)
          Rings: 0
  NativeFormats: 0x2 (gsm)
    WriteFormat: 0x40 (slin)
     ReadFormat: 0x40 (slin)
 WriteTranscode: Yes
  ReadTranscode: Yes
1st File Descriptor: 39
      Frames in: 0
     Frames out: 0
 Time to Hangup: 0
   Elapsed Time: 0h0m0s
  Direct Bridge:
Indirect Bridge:
 --   PBX   --
        Context: xxxxxxxx
      Extension: xxxxxxxx
       Priority: 2
     Call Group: 0
   Pickup Group: 0
    Application: Dial
           Data: SIP/1xxxxxxxxxx
    Blocking in: ast_waitfor_nandfds
      Variables:

SYSTEMSTATUS=SUCCESS
SIPCALLID=146ef044580a3a795de94b541db21249@xxx.xxx.xxx.xxx
SIPUSERAGENT=Asterisk PBX
SIPDOMAIN=xxx.xxx.xxx.xxx
SIPURI=sip:300@85.214.85.180

  CDR Variables:
level 1: clid=xxxxxxxx
level 1: src=xxxxxxxx
level 1: dst=xxxxxxxx
level 1: dcontext=xxxxxxxx
level 1: channel=SIP/300-b6b241c0
level 1: dstchannel=SIP/xxxxxxxx
level 1: lastapp=Dial
level 1: lastdata=SIP/xxxxxxxx
level 1: start=2009-09-21 11:59:34
level 1: answer=2009-09-21 11:59:34
level 1: end=2009-09-21 11:59:34
level 1: duration=0
level 1: billsec=0
level 1: disposition=NO ANSWER
level 1: amaflags=DOCUMENTATION
level 1: uniqueid=1253548774.7688

So what occurred here? The who, what, where and how? The four most important variables are easy to parse:


How: Name: SIP/300-b6b241c0
Where: DNID Digits: 002314599711
What: SIPUSERAGENT=Asterisk PBX
Who: SIPURI=sip:300@85.214.85.180

How? An attacker leveraged a SIP account on the honeypot (username=300) Where? They tried dialing 16085541772 What? They likely tried trunking the SIP account to their Asterisk box Who? 85.214.85.180 So who owns the server? A company reselling hosting:


inetnum:        85.214.16.0 - 85.214.139.255
netname:        STRATO-RZG-DED2
descr:          Strato Rechenzentrum, Berlin
country:        DE
admin-c:        SRDS-RIPE
tech-c:         SRDS-RIPE
remarks:        ************************************************************
remarks:        * Please send abuse complaints to abuse-server@strato.de   *
remarks:        * or fax +49-30-88615-755 ONLY.                            *
remarks:        * Abuse reports to other e-mail addresses will be ignored. *
remarks:        ************************************************************
status:         ASSIGNED PA
mnt-by:         STRATO-RZG-MNT
source:         RIPE # Filtered

Wondering if this is spoofed (since some seem to think it is):


# nmap -sU -p 5060 85.214.85.180

Interesting ports on server1.stacheshof.de (85.214.85.180):
PORT     STATE         SERVICE
5060/udp open|filtered sip

Nmap finished: 1 IP address (1 host up) scanned in 1.755 seconds

Likely not spoofed however, this does not stop someone from compromising a host and using it to do other things. The likelihood of this server being compromised AND running a PBX on it is low and here is my reasoning for this comment, I’ve seen this address quite a few times connecting to different addresses during different months. I would think a server’s administrator would notice a PBX server running on their system. What makes things even more interesting is the following, the attacker tries to make one call, then starts changing his (or her) dialing patterns:


002314599711 - 20100921-11:41:33
92314599711 - 20100921-11:47:42
9002314599711 - 20100921-11:54:31
90112314599711 - 20100921-12:00:29

The calls were placed at odd times to make me think they were automated. The first call was at 11:41 followed by six minute gap, followed by a 7 minute gap, followed by another 6 minute gap. In previous times, automation was instant. Even odder is, how is this person aware that a call didn’t go through and that they should be changing the dial plan? My response is, they have some form of visibility into that destination number. Calls from my honeypot sound just as any other call with the exception they all end up in one place. Voicemail to a non-existent account. This further allows me to hear the audio if any is left.

After accumulating all of this data, I figured it would be best to share it with others in the industry via the “VoIP Abuse Project.” The purpose of the project is to create a blacklist of sorts for any engineer or admin to be able to use to block out the bad guys. Alongside blocking out the bad guys, it always offers a place to showcase the realities of “abuse desks” of companies whose networks are being used in attacks. That goal too is simple, systems are attacked and reported, companies responding and correcting issues should be applauded, companies ignoring the issue should know that it will only be a matter of time before their clients cost them more in lost revenue. It’s their networks, they should be more responsible about what leaves their network. Anyhow, this has become rather long so I will leave this as a part one. Stay tuned for more.