Monthly Archives: August 2008

Update on the Aircell / VoIP-on-a-plane prohibition – and an Aircell response

After my two posts on Tuesday explaining how Aircell was probably blocking VoIP and then why the Phweet/Tringme worked (temporarily), there have been a number of other posts that should be mentioned here:

  • Om Malik posted “Aircell: On U.S. Planes, VoIP Will Be Muted” where he relayed a conversation with an Aircell spokesperson that included this classic quote: “we are doing our best to make VoIP services unusable.” Some of the comments on Om’s post are quite good, too.

    Om also relayed that Aircell indicated that Phweet/TringMe will no longer work on their service. As I expected, they blocked the traffic pattern of the service. Aircell will engage in the Whack-A-VoIP-Call and over time they will build up an increasing number of patterns that they will block.

  • The folks at Tringme posted “TringMe Conversations (Phweet, Aircell & TringMe Traffic Patterns)” which includes this interesting part (my emphasis added):

    TringMe uses TCP and it was a conscious decision. We developed a sophisticated congestion control and packet handling algorithms which allowed us to achieve the advantage of UDP over a reliable TCP connection at good extent. As Dan and others would have noticed, we send traffic in varying small and larger blocks depending on network conditions which is way different from a typical VoIP traffic patterns. This kind of pattern was not meant to break any VoIP blockages, however the goal was to get the best quality even on slower or congested links & we were able to meet the design goal successfully.

    What is interesting here to me is that they do vary the pattern that they use. While this certainly could happen with other VoIP solutions depending upon codec selection, etc., I haven’t seen much variance in actual deployments (outside of Skype, which does vary according to how it is punching through a firewall). The TringMe folks go on to talk about the advantages of TCP (with of course a bit of a sales pitch for why you might want to use their client).

  • Dean Foust at Business Week posted “You CAN make VOIP calls on airplanes. Joy.” expressing his curmudgeonly view that VoIP-on-a-plane was not at all desirable. (As I said in P.S. to my 2nd post, I definitely think there are larger societal/cultural issues that we need to work out about whether or not VoIP on a plane is something we really want to have.)

  • Irwin Lazar posted on the Enterprise 2.0 blog “American Airlines Aircell Reaction” and made the comment:

    Dan York suspects that this will lead to an arms race as Aircell fights VOIP users. I think he’s right, but I don’t think more than a handful of users will care enough to fight the battle.

    To a certain degree, Irwin’s probably right. Some small % of users will actually care enough to try to find ways around Aircell’s blocking. Most folks won’t and will just accept that they can’t use VoIP. Irwin also has some good comments about the impacts of VoIP availability on virtual workers and eliminating the “escape from the office” time that flyers have today. (Some people will want to be “always-on” while others will not.)

  • Andy Abramson posted “From the Department of Stupid Is as Stupid Does-Aircell” pointing out that Aircell sells VoIP solutions for airplanes and thus it is a bit strange to see them also saying “No VoIP on planes!” (Although their position is that this is the policy of their customer, American Airlines.)

AIRCELL RESPONDS

Speaking of Aircell’s position, I did receive a nice note from someone with Aircell’s PR firm that stated this:

Thanks for your informative posts about Aircell and the use of VoIP on Gogo today – good reading. We’ve been asked by many outlets for an official response to the VoIP attempts on Gogo so in case you are interested, here is Aircell’s official statement:

It is against American’s policy and Gogo’s terms of service to use VoIP. Aircell has multiple protocols and practices in place to prevent the use of VoIP. Obviously, it is extremely difficult to stop every instance of VoIP but Aircell is monitoring and working constantly to enforce American’s policy and Gogo’s terms of service.

To a certain degree, this statement reminds you that at the end of the day Aircell is simply the service provider implementing the policy of the customer. So it’s really the American Airlines policy that has the VoIP prohibition….. but….

I could go along with that except for one minor little detail. The
Gogo Terms of Service very clearly state the VoIP prohibition. The ToS also states very clearly that the Gogo Inflight Internet Service is provided by Aircell LLC. There is no mention at all in the ToS of American Airlines. Is this Gogo service provided ONLY to American Airlines? Was it created for only American Airlines? Will it not be sold to any other airline?

If it was only created for American Airlines and will only be used by American flights, then sure, the Gogo ToS line up with the customer’s policy. If this is intended to be a generic service and American just happens to be the first customer then I think it’s a bit unfair for Aircell only to be pointing to American. It is, after all, Aircell’s ToS.

But perhaps that’s getting too far down in the semantic weeds…

Technorati Tags:
, , , , ,

Blue Box Podcasts #81 and #82 now available for download…

MD_bluebox157-2.jpgAfter a long hiatus, I’m finally starting to get Blue Box episodes flowing again. I’ve just put up two in the past two days:

And I have some more special editions I’m looking to put out soon. This summer was a bit chaotic for me with a physical move from Vermont to New Hampshire, but I’m hoping things are now settled down enough that I can get back into regular production of these episodes….

Technorati Tags:
, , , , , , , ,

The reason why (probably) you can use Phweet on a plane when Skype is blocked

Update #1: Since publishing this post this morning, I’ve learned of David Berlind’s success using AOL’s video chat from the plane. Odds are that it works for a similar reason to what I outline for Phweet below (and that eventually it will be blocked).

Update #2: As expected, GigaOm is now reporting that Aircell is blocking the Tringme VoIP client used by Phweet.

So how did Andy Abramson make a call using Phweet on the new Aircell Gogo Internet service on American Airlines planes? Why was the call not blocked like Skype or Sightspeed calls were?

As I discussed in my last post, VoIP calls have certain network characteristics that make them easy to identify and, in Aircell’s case, block. A VoIP call starts up… a stream of typically 100 packets per second start streaming over UDP between two endpoints… the networking monitoring software notices this, watches it for several seconds, decides it’s VoIP, and blocks the stream.

So why was Phweet’s web-based solution different? Why did it work when Skype, Sightspeed and other services fail?

THE CASE OF THE MISSING PROTOCOL

To understand, we need to look at the traffic from the Flash softphone that is being used by Phweet. The Flash-based softphone is actually from Tringme and is integrated into Phweet’s services. Let’s look at the network capture from a call I had today with Phweet CEO Stuart Henshall using the Flash softphone:

voip-wireshark-phweet1.jpg

Like the Skype call in my other post, you can see that there is a jump here (at about the 155 second mark) where the Tringme client connects to the Phweet conference bridge. We now have streaming audio, coming in a little less than 100 packets per second (more on that later). Stuart and I talked too long so I can’t easily show you the corresponding dropoff when the call ends (and still use the same scale as my other blog post) but suffice it to say that when the call is over the traffic returns to the previous lower (and bursty) level. Like the Skype calls, the packets sent by the Tringme softphone were small.

So if the pattern is the same, why didn’t Aircell’s sniffers recognize the call and block it? (Have you figured it out yet?)

To see one of my guesses, recall that in my last post I showed how a Skype call was typically all UDP:

voipcall-wireshare-udpred-1.jpg

Now let’s look at the Phweet call with the exact same UDP display filter turned on in Wireshark:

voip-wireshark-phweet2.jpg

Oops! Where’s the UDP?

Exactly.

If I look at the traffic and set a display filter to show all the traffic to the IP address of Phweet’s conference bridge, I can see (blue line) that indeed the traffic to/from Phweet is responsible for the uptick in packets:

voip-wireshark-phweet3.jpg

So if you haven’t already figured it out, let’s bring in a graph showing the other transport protocol, TCP, in green:

voip-wireshark-phweet4.jpg

And there you go… one very possible reason why Phweet may work and Skype, Sightspeed and others were blocked is simply this:

The Tringme Flash-based softphone is sending audio over TCP and Aircell is not recognizing and blocking VoIP calls over TCP.

Or at least Aircell wasn’t blocking TCP. (They probably are by now or will be soon.)

Now to be fair, if Aircell isn’t blocking TCP, this was probably a decent assumption to make. I mean, the typical mindset to date has been… who in their right mind would send audio streams over TCP?

In all the VoIP systems I’ve worked with, I can’t think offhand of any other systems that send audio over TCP. As part of its range of tricks to get through firewalls, I understand that Skype can use TCP if it is unable to send over UDP, but I’ve never captured it doing so. The IP-PBXs I’m familiar with, both commercial and open source, all send RTP over UDP.

The reason is simple… speed. Setting up a TCP connection involves the “three-way-handshake” and each packet contains information to ensure the packet will get there. If packets aren’t received by the other end, there are requests for retransmission. Data is guaranteed to get there and get there in the right order. It’s perfect for transfers of data where you want to ensure it gets there. However, in poor network conditions, it can be slower due to frequent retransmissions.

UDP on the other hand, makes no guarantees of delivery. Packets are just tossed out on the network pointed at the other endpoint and you hope that they get there. It’s fast, because there’s no involved connection setup, no packet tracking and no retransmissions. If some packets get lost along the way, oh well. Most VoIP systems tend to use UDP because the human ear is okay with a certain degree of lost audio. If some packets get lost along the way, we can probably still make out what the other person is saying. (And cell phones have gone far to degrade our perception of audio quality… but that’s another rant.)

Now in a network with good bandwidth, the performance between UDP and TCP may be similar enough to be okay… but in poor networks, TCP will be slower. For that reason, most folks in the VoIP industry have opted to go with UDP.

So one explanation for why Phweet/Tringme worked on a plane when Skype didn’t is that Aircell made an assumption that all VoIP calls that they want to block would send audio over UDP. The Tringme softphone used TCP instead – and so it got through.

THRESHOLD TOO HIGH?

However, the network trace gives a second possibility for why Phweet worked by Skype didn’t. If you look at the capture from my last post, you can see that the Skype call was coming across right about 100 packets per second:

voipcall-wireshark-udponly.jpg

The math behind this is fairly straightforward. If you are using the 20ms sampling rate common in VoIP systems, that means that there are 50 packets per second (simple way to figure this out – divide 1 second by 0.02 to figure out how many times 20ms is in 1 second). With two audio streams (sending and receiving), the Skype call would generate roughly 100 packets per second.

Now look at the Phweet call using the Tringme Flash softphone:

voip-wireshare-phweet5.jpg

You can see that the packet rate is not at 100 packets per second and is more somewhere between 60 and 70 packets per second. There’s a simple possible explanation here:

The Tringme Flash client could be using 30ms sampling.

If you increase the size of each audio sample from 20ms to 30ms, you reduce the number of packets generated per second from 50 to… 33. (1 divided by 0.03 to get how many 30ms packets are in 1 second) Double that for the sending and receiving audio streams and you have a total audio packet flow of around 66 packets per second. Which looks awfully close to what is here.

So a second potential reason why Phweet worked and Skype didn’t could be this:

Aircell has set their threshold too high for recognizing all VoIP calls.

It could be that their software is set to trigger on a stream of small packets between two endpoints at 80 packets per second for a period of, say, 5 seconds or more. Anything that hits that rule gets blocked. Maybe it’s 90 packets per second. Maybe it’s 70.

The point is that Aircell may have made the assumption that all VoIP clients would do 20ms sampling and generate a packet flow of ~100 packets per second. The Tringme client used by Phweet does 30ms sampling and so comes in below this threshold.

THE FUTILITY OF FIGHTING THIS ARMS RACE

Now there may yet be another reason why Aircell didn’t detect and block the Tringme softphone used by Phweet. But I would hope that the two potential explanations above would point out the inherent futility in Aircell’s attempts to block VoIP:

There will always be some VoIP client that will find a way around Aircell’s blocks.

If Aircell is blocking on pattern recognition as I’ve discussed in this post and the previous one, someone will make a VoIP client that goes to 40ms sampling over TCP, dropping the packet flow down to a total 50 packets per second. Maybe they’ll use a codec that generates larger packets so that voice traffic becomes indistinguishable from file traffic. Maybe they’ll create a VoIP client that is easily configurable by the person using it so that someone can tweak the parameters while they are on the plane (and subsequently twitter/blog about how they did it).

If they are blocking on protocols like SIP, someone will start using an IAX2 softphone… and when they block that, someone will resurrect an H.323 softphone… and so on.

If they start doing deeper packet inspection, someone will figure out how to use a VPN client that encrypts everything and perhaps fills out all packets so that they are an identical size.

There are a zillion things that people can do to route around and defeat blocking… and in this age of social media, such successes get spread via wildfile (induced by services/sites like Twitter, Techmeme and others). And so many more will learn how to get around Aircell’s blocking… until Aircell improves their blocking… until someone gets around that blocking… and Aircell improves their blocking… and someone gets around that blocking…

Aircell can fight that battle. From their ToS and comments made after this incident, it seems that they will fight this battle and waste their time, energy and resources trying to fight the never-ending cycle of creativity by those who want to go around the system.

Good luck to them.

P.S. I do think there are larger cultural issues we need to work out with regard to having access to VoIP in a plane. I do agree with Aircell’s CEO’s comment (in Joanna Stern’s article) that most all travelers have an aversion to people talking loudly on a plane. And yes, access to VoIP calls could definitely make this worse. (I recently tweeted about my experience with cell phones on the train.) But I think we need to figure out some way to deal with that outside of blocking. As Andy said on yesterday’s Squawk Box when we talked about all of this, perhaps there will be new sections on planes where people will be free to talk (or “quiet-zone” sections where they aren’t!).

Technorati Tags:
, , , , , , , ,

Dan York is Best Practices Chair for the VoIP Security Alliance and writes here and at DisruptiveTelephony. You can follow him on Twitter or identi.ca.

How Aircell is (probably) blocking VoIP phone calls on planes (hint… VoIP Whack-A-Mole)

aircell-gogo-logo.jpgHow is Aircell blocking VoIP phone calls from systems like Skype, SightSpeed and Gizmo? (And how did Andy get through with Phweet?)

Ever since last week’s announcement of the “Gogo Inflight Internet Service” provided to American Airlines by a company called Aircell – and the ensuing coverage in the blogosphere – I’ve been getting asked about how exactly Aircell is blocking VoIP calls. Especially after Andy Abramson was able to make a call using Phweet. Aircell is very clear in the Gogo terms of service that no voice calls are allowed:

No Voice Applications. You will not use any type of voice application (including, without limitation, voice over Internet protocol) without written permission from Aircell.

And early users of the system who tried VoIP calls reported that indeed after about 5 seconds or so, their VoIP conversation was terminated. Repeatedly. They could use Skype, for instance, for IM, but not for voice.

So how is Aircell blocking VoIP? (And how did Andy get through with Phweet?)

Unfortunately, I was a wee bit busy last week when all this was breaking, so it’s taken me until now to come up for air enough to write about how Aircell could be blocking VoIP calls on the planes.

THE BIG CAVEAT

First, I should state up front – I have absolutely NO connection to Aircell, Gogo, American Airlines, etc.. I don’t know exactly how they are blocking VoIP calls but am laying out here how they could be blocking VoIP calls.

[I also feel compelled to say that I personally think it’s silly of Aircell to block VoIP because there will inevitably be people who figure out ways to route around the blocking. I think Aircell’s excuse that they want to block people from talking loudly is also rather lame. I’ve been on long flights where I was trying to sleep and I’ve had two people talking very loudly in the seat behind me or next to me. (And yes, sometimes I’ve asked them to please speak quieter.) Other times I’ve had babies screaming and crying for most of the flight… or other “energetic” children carrying on. No, I don’t really want my neighbor to be in an involved VoIP call but my point is that there are disruptions already. Part of me wonders if there aren’t really more technological issues with doing streaming audio/video, but anyway… that’s their policy and if you want to use their service, you have to agree. You don’t (yet) have a choice in services to use, and you probably won’t.]

LET’S PLAY “SPOT THE VOIP PHONE CALL”

Given that Skype conversations in particular are encrypted, I’ve had several people ask me if this means that Aircell can decrypt Skype calls. How else, they ask, can Aircell differentiate between Skype text chat and Skype voice chat? It’s simple really:

Pattern recognition.

A VoIP call in progress has a distinct profile from a network packet point-of-view. In general, the audio streams of VoIP calls (as compared to the call control/signalling channels) have the following characteristics:

  1. there are a zillion small packets
  2. the packets are sent over UDP versus TCP
  3. the packets are sent using the Real-time Transport Protocol (RTP) over UDP

Now with Skype’s encryption, network software can’t know about the contents of the audio stream, so the software can’t know about my #3 here, but the software can recognize the pattern based on the first two. For other non-encrypted services, the software can very simply look for RTP streams.

Now, why do VoIP systems use a zillion small packets? Typically, a VoIP system will sample the speaker’s voice at an interval of every 10, 20 or 30 milliseconds. Most I’m familiar with seem to go for a 20ms rate. So 20ms of audio is captured, encoded digitally and sent off in an IP packet. The exact size of the IP packet will vary depending upon what codec is used to encode the audio. The standard G.711 packets will be at one size and G.729 will be much smaller. Many of the VoIP streams I’ve captured in network traces have a total packet length of somewhere between 35-70 bytes.

To put that in perspective, understand that an Ethernet packet can have typically a max size of 1500 bytes. And packets sent by various protocols can be even larger and will be “fragmented” into smaller pieces (for instance, 1500-byte pieces) to be moved across the network. [Network geeks: Please give me some poetic license here… I realize I could be more precise, but I’m trying not to completely bore the readers.]

The point is that voice packets are typically tiny in comparison to other packets – and there are a lot of them.

How many? Well, if you take a 20ms sampling rate, that means that you are sampling the audio voice 50 times each second… so that’s 50 packets per second for one audio stream. Almost every voice conversation involves two audio streams (one from the caller, one from the recipient) and so you are looking at 100 packets per second for a typical two-way VoIP conversation.

The reason for this is relatively simple. In a file transfer, you are looking to move a file across the network as fast as possible – but you aren’t necessarily in “real-time”. So you generally will stuff the packets with as much info as you can and push them across the network. With voice, we are making use of the fact that the human ear will deal with some lost audio, and so we are chopping up the audio up into a zillion tiny pieces, tossing them in unreliable UDP packets and hoping that enough get there so that the listener can make sense of the conversation.

Put another way, if I wanted to send you this blog post via snail-mail, I could print it out, stick it in an envelope and mail it to you. That’s file transfer. On the other hand, if I were to chop this blog post up and write each word on a post card and stick them in the snail-mail to you, odds are that enough would arrive at the other end that you could assemble them into something like this post. (With luck, maybe you could even make the post shorter!)

That’s VoIP.

So once you know the pattern, it’s fairly easy to spot the calls. For instance, where’s the VoIP call in this network trace?

voipcall-wiresharktrace-raw-1.jpg

Do you see it? How about this trace with a filter turned on to show UDP in red?

voipcall-wireshare-udpred-1.jpg

Ta da… there’s your VoIP call!

Let’s look at this one in a bit more detail:

voipcall-wireshark-udponly.jpg

There it is… all in UDP… and coming in at about 100 packets per second. And if I look at the actual Wireshark traces, I can see that these 100 packets per second are all very tiny sizes. Many of them are between 37 and 50 bytes.

And this is an encrypted Skype call!

No need to decrypt it. Just see that it’s a steady stream of 100 very small packets per second (50 packets per second each way) all over UDP.

Kill the stream. Block it. Conversation dead. No more VoIP on the plane.

It’s basically the network security version of Whack-A-Mole. See a VoIP stream start up… block it. See another one… block it. See yet another… block it. Whenever anything pops up that meets the profile, stomp on it.

This explains, too, why people could talk for a few seconds and then had their conversations terminated. The pattern has to appear in the network monitoring software. The software has to be sure it’s a VoIP stream and not something else… and then the software can block it.

Now I don’t know for a fact that this is how Aircell is blocking VoIP, but it would be easy enough to do it this way.

GETTING MORE SOPHISTICATED

There are, of course, easier ways to kill the conversations. If the VoIP calls use unencrypted audio streams then it’s incredibly trivial to block. Just block the RTP protocol. Period. End-of-story. Now this does involve a hair more packet inspection in that the software has to look farther into the packet headers to see the protocol type, but again this is easy to do. All RTP is typically used for is streaming audio or video… block it and the “problem” goes away.

They could of course go even further into packets and see if they are Session Initiation Protocol (SIP) packets and if so, what the packets are asking to do. If one endpoint sends a SIP INVITE packet to another endpoint and indicates that an audio conversation is to start, the software could again simply block the impending audio stream. (Of course this couldn’t be done if the SIP was encrypted…)

The software could also simply block ports. Block any usage of port 5060 or 5061 and you would probably kill off most “regular” SIP conversations. (Yes, SIP endpoints can make connections on non-standard ports, but the majority of clients probably wouldn’t.) The challenge here is that some SIP endpoints also would use SIP to set up non-audio communication channels like text chat, so blocking all SIP would also block SIP-based text chat which is probably not desirable.

The software could also block on a service level… if it knew, for instance, the host names or IP addresses for the media servers and consumer services (through which the audio would be sent), the software could block all connections to those media servers.

There’s a whole range of additional layers the network monitoring software could use. Any good system will have a “defense in depth” strategy and make use of many of these different algorithms.

Of course, adding on these layers does require more computing power and will undoubtedly add some latency (even on a microscopic level). It may be that for right now they can simply do the pattern recognition approach and shut down VoIP calls.

SO HOW DID ANDY DO IT?

Okay, so how did Andy make a call using Phweet? Given that this post has already gone on this long, I’ll publish my guess in a subsequent post. The text above should give you enough clues, though… any pattern recognition system is inherently fragile because it depends upon recognizing patterns. So what if the audio packets you are sending don’t match any known patterns? What if (hint) the folks at Aircell forgot to watch all protocols?

Stay tuned for some more network charts in my next post… 🙂

P.S. This is, by the way, why I think that these type of systems trying to block VoIP calls are inherently doomed… someone will inevitably find a way to “cloak” their VoIP calls so that they are unrecognizable or indistinguishable from other data traffic… it’s a cat and mouse game and inevitably people will find ways to get around the watchers…

Technorati Tags:
, , , , , ,

Dan York is Best Practices Chair for the VoIP Security Alliance and writes here and at DisruptiveTelephony. You can follow him on Twitter or identi.ca.

Oyster not all it’s Cracked-up to be

Some Dutch researchers recently illustrated once again that “security by obscurity” is not a good way to secure systems. Transport for London (TFL) have for some years been running a prepay card system called the “Oyster Card”. The Oyster Card is an RFID card that you wave over a sensor at the start and end of your train trip, which takes money from your prepay account with TFL. Oyster offers a preferential rate, cheaper than paper tickets, and has had a very high take-up rate with London commuters and residents.

However, the system has now been cracked. At the heart of the Oyster card is a chip called the Mifare Classic, which uses a secret algorithm. Some Dutch researchers decided to target this system, and have discovered how this works. As a demonstration, they used their knowledge to create their own card, which they used to travel around free on the London Underground for a day.

Their interest in the Oyster Card probably stems from the fact that the same Mifare Classic chip is also used in access cards used to secure Government buildings in the Netherlands. In a rather nice demonstration of the separation of powers working in a democratic country, they now have leave from a Dutch judge to publish details of how the Mifare algorithm works, which they will be doing at a security conference in the coming October. This would not have been the outcome that the Dutch government would want, since they now have to take extra steps to secure buildings with more security personnel.

But “hushing the problem up” is not a solution in the security world. The problems don’t go away when you punish the researchers. For every ethical hacker there are probably another two Black-Hats who want to sell the information to those that could profit from free travel in London, or access to Dutch Government property.