Update #1: Since publishing this post this morning, I’ve learned of David Berlind’s success using AOL’s video chat from the plane. Odds are that it works for a similar reason to what I outline for Phweet below (and that eventually it will be blocked).
Update #2: As expected, GigaOm is now reporting that Aircell is blocking the Tringme VoIP client used by Phweet.
So how did Andy Abramson make a call using Phweet
on the new Aircell Gogo Internet service on American Airlines planes? Why was the call not blocked like Skype or Sightspeed calls were?
As I discussed in my last post, VoIP calls have certain network characteristics that make them easy to identify and, in Aircell’s case, block. A VoIP call starts up… a stream of typically 100 packets per second start streaming over UDP between two endpoints… the networking monitoring software notices this, watches it for several seconds, decides it’s VoIP, and blocks the stream.
So why was Phweet’s web-based solution different? Why did it work when Skype, Sightspeed and other services fail?
THE CASE OF THE MISSING PROTOCOL
To understand, we need to look at the traffic from the Flash softphone that is being used by Phweet. The Flash-based softphone is actually from Tringme and is integrated into Phweet’s services. Let’s look at the network capture from a call I had today with Phweet CEO Stuart Henshall using the Flash softphone:
Like the Skype call in my other post, you can see that there is a jump here (at about the 155 second mark) where the Tringme client connects to the Phweet conference bridge. We now have streaming audio, coming in a little less than 100 packets per second (more on that later). Stuart and I talked too long so I can’t easily show you the corresponding dropoff when the call ends (and still use the same scale as my other blog post) but suffice it to say that when the call is over the traffic returns to the previous lower (and bursty) level. Like the Skype calls, the packets sent by the Tringme softphone were small.
So if the pattern is the same, why didn’t Aircell’s sniffers recognize the call and block it? (Have you figured it out yet?)
To see one of my guesses, recall that in my last post I showed how a Skype call was typically all UDP:
Now let’s look at the Phweet call with the exact same UDP display filter turned on in Wireshark:
Oops! Where’s the UDP?
If I look at the traffic and set a display filter to show all the traffic to the IP address of Phweet’s conference bridge, I can see (blue line) that indeed the traffic to/from Phweet is responsible for the uptick in packets:
So if you haven’t already figured it out, let’s bring in a graph showing the other transport protocol, TCP, in green:
And there you go… one very possible reason why Phweet may work and Skype, Sightspeed and others were blocked is simply this:
The Tringme Flash-based softphone is sending audio over TCP and Aircell is not recognizing and blocking VoIP calls over TCP.
Or at least Aircell wasn’t blocking TCP. (They probably are by now or will be soon.)
Now to be fair, if Aircell isn’t blocking TCP, this was probably a decent assumption to make. I mean, the typical mindset to date has been… who in their right mind would send audio streams over TCP?
In all the VoIP systems I’ve worked with, I can’t think offhand of any other systems that send audio over TCP. As part of its range of tricks to get through firewalls, I understand that Skype can use TCP if it is unable to send over UDP, but I’ve never captured it doing so. The IP-PBXs I’m familiar with, both commercial and open source, all send RTP over UDP.
The reason is simple… speed. Setting up a TCP connection involves the “three-way-handshake” and each packet contains information to ensure the packet will get there. If packets aren’t received by the other end, there are requests for retransmission. Data is guaranteed to get there and get there in the right order. It’s perfect for transfers of data where you want to ensure it gets there. However, in poor network conditions, it can be slower due to frequent retransmissions.
UDP on the other hand, makes no guarantees of delivery. Packets are just tossed out on the network pointed at the other endpoint and you hope that they get there. It’s fast, because there’s no involved connection setup, no packet tracking and no retransmissions. If some packets get lost along the way, oh well. Most VoIP systems tend to use UDP because the human ear is okay with a certain degree of lost audio. If some packets get lost along the way, we can probably still make out what the other person is saying. (And cell phones have gone far to degrade our perception of audio quality… but that’s another rant.)
Now in a network with good bandwidth, the performance between UDP and TCP may be similar enough to be okay… but in poor networks, TCP will be slower. For that reason, most folks in the VoIP industry have opted to go with UDP.
So one explanation for why Phweet/Tringme worked on a plane when Skype didn’t is that Aircell made an assumption that all VoIP calls that they want to block would send audio over UDP. The Tringme softphone used TCP instead – and so it got through.
THRESHOLD TOO HIGH?
However, the network trace gives a second possibility for why Phweet worked by Skype didn’t. If you look at the capture from my last post, you can see that the Skype call was coming across right about 100 packets per second:
The math behind this is fairly straightforward. If you are using the 20ms sampling rate common in VoIP systems, that means that there are 50 packets per second (simple way to figure this out – divide 1 second by 0.02 to figure out how many times 20ms is in 1 second). With two audio streams (sending and receiving), the Skype call would generate roughly 100 packets per second.
Now look at the Phweet call using the Tringme Flash softphone:
You can see that the packet rate is not at 100 packets per second and is more somewhere between 60 and 70 packets per second. There’s a simple possible explanation here:
The Tringme Flash client could be using 30ms sampling.
If you increase the size of each audio sample from 20ms to 30ms, you reduce the number of packets generated per second from 50 to… 33. (1 divided by 0.03 to get how many 30ms packets are in 1 second) Double that for the sending and receiving audio streams and you have a total audio packet flow of around 66 packets per second. Which looks awfully close to what is here.
So a second potential reason why Phweet worked and Skype didn’t could be this:
Aircell has set their threshold too high for recognizing all VoIP calls.
It could be that their software is set to trigger on a stream of small packets between two endpoints at 80 packets per second for a period of, say, 5 seconds or more. Anything that hits that rule gets blocked. Maybe it’s 90 packets per second. Maybe it’s 70.
The point is that Aircell may have made the assumption that all VoIP clients would do 20ms sampling and generate a packet flow of ~100 packets per second. The Tringme client used by Phweet does 30ms sampling and so comes in below this threshold.
THE FUTILITY OF FIGHTING THIS ARMS RACE
Now there may yet be another reason why Aircell didn’t detect and block the Tringme softphone used by Phweet. But I would hope that the two potential explanations above would point out the inherent futility in Aircell’s attempts to block VoIP:
There will always be some VoIP client that will find a way around Aircell’s blocks.
If Aircell is blocking on pattern recognition as I’ve discussed in this post and the previous one, someone will make a VoIP client that goes to 40ms sampling over TCP, dropping the packet flow down to a total 50 packets per second. Maybe they’ll use a codec that generates larger packets so that voice traffic becomes indistinguishable from file traffic. Maybe they’ll create a VoIP client that is easily configurable by the person using it so that someone can tweak the parameters while they are on the plane (and subsequently twitter/blog about how they did it).
If they are blocking on protocols like SIP, someone will start using an IAX2 softphone… and when they block that, someone will resurrect an H.323 softphone… and so on.
If they start doing deeper packet inspection, someone will figure out how to use a VPN client that encrypts everything and perhaps fills out all packets so that they are an identical size.
There are a zillion things that people can do to route around and defeat blocking… and in this age of social media, such successes get spread via wildfile (induced by services/sites like Twitter, Techmeme and others). And so many more will learn how to get around Aircell’s blocking… until Aircell improves their blocking… until someone gets around that blocking… and Aircell improves their blocking… and someone gets around that blocking…
Aircell can fight that battle. From their ToS and comments made after this incident, it seems that they will fight this battle and waste their time, energy and resources trying to fight the never-ending cycle of creativity by those who want to go around the system.
Good luck to them.
P.S. I do think there are larger cultural issues we need to work out with regard to having access to VoIP in a plane. I do agree with Aircell’s CEO’s comment (in Joanna Stern’s article) that most all travelers have an aversion to people talking loudly on a plane. And yes, access to VoIP calls could definitely make this worse. (I recently tweeted about my experience with cell phones on the train.) But I think we need to figure out some way to deal with that outside of blocking. As Andy said on yesterday’s Squawk Box when we talked about all of this, perhaps there will be new sections on planes where people will be free to talk (or “quiet-zone” sections where they aren’t!).
air travel, aircell, airlines, voip, security, voip security, gogo, american airlines, skype
Dan York is Best Practices Chair for the VoIP Security Alliance and writes here and at DisruptiveTelephony. You can follow him on Twitter or identi.ca.