Last night, millions of people across the eastern seaboard found themselves unable to go online and access the Internet. The culprit? A Comcast DNS outage that lasted more than three hours and affected customers from Boston to Baltimore. These kinds of attacks can hit anyone, including us. And it’s likely that if an attack was large enough to disrupt Comcast, it could be large enough to disrupt us. That’s scary.

When we launched introduced OpenDNS more than four years ago, our promise was this: the fastest, most reliable DNS service available. Since then we’ve added features and built out enhancements including malware protection, Web content filtering and SmartCache. That said, offering ultra-reliable DNS service is still at the core of what we do. This is part of the reason why we added a new datacenter in Singapore recently.

Over the past four+ years, we have been fortunate to have a perfect, 100 percent uptime record and we will work hard to maintain that. As we saw with Comcast last night, even great ISPs have outages when attacked with massive amounts of malicious traffic. This is why we will continue to add capacity, far in excess of what we actually need. The real solution to this is better security for end-users so they don’t get infected and become vehicles for DDoS attacks.

So, if you just got set up with OpenDNS last night, welcome! Hopefully last night was the last time you’ll ever be without Internet due to a DNS issue. We’ll work hard to make sure it was. If you’ve been set up with OpenDNS for a while now, you probably didn’t notice there was any issue at all.

The team here has been hard at work over the past few months getting a new datacenter set up, and I’m happy to report that as of today, our Singpore datacenter is online and serving production OpenDNS traffic. The Singapore server marks our 12th datacenter globally and the first of a number planned for Asia.

One of the benefits of OpenDNS is that we use a technique called Anycast routing in how we run our network. Anycast means that no matter where you are in the world, your DNS requests route through our closest datacenter. And when we do maintenance that requires us to take a site offline, our routing topology ensures you will route to our next closest datacenter. It also means that when we bring up a new datacenter that is closer to you, your DNS requests will automatically start routing to it. So for the bulk of our users in Singapore, Thailand, Vietnam, India and throughout Asia, this new Singapore datacenter promises an even faster Internet.

And here’s a photo of what our installation looks like as it was being racked:

This Singapore datacenter is only the start. We’re planning on adding a new datacenter in Frankfurt, Germany in early 2011. After that, the plan is to continue expanding our footprint from there. As always, you can take a look at our global system status on our Systems page.

PS — Internet routing is not a perfect science and requires a lot of work to get right. If you are in Asia and a traceroute doesn’t show you talking to Singapore for DNS we want to know! Please send a traceroute from your computer to 208.67.222.222 to our support department so we can see which networks in Asia aren’t seeing our new routes.

A standardized speed test for OpenDNS would be difficult since physical locations, ISPs and other setup factors vary. That’s why we value independent speed testing of OpenDNS so much.

Mayank Sharma of Newsforge, the online newspaper for Linux and open source, did some testing of his own - in India – and found that OpenDNS loads pages “much more quickly” than his ISP.

I tested that claim from my home base in India. After switching to OpenDNS, content-laden Web sites like news.com, cnn.com, bbcworld.com, and myspace.com loaded a lot more quickly, ping times were considerably lower, and query response times (measured with dig -x site ) to news.com, lxer.com, osnews.com, distrowatch.org, and bbcworld.com were lower by 10 to 25% compared to times when I was using my ISP’s DNS.

That shoots holes in the idea that OpenDNS is only faster if you’re in the U.S. or U.K.

It’s been a long time coming, but we are now online and operational in London! We actually turned up our routing announcements about two days ago but I wanted to hold off on the blog post to make sure everything was stable. Some folks in the forums noticed we were online and beat me to the announcement.

I’ve been using a server in Amsterdam, hosted by my friend Peter, to test how latency changed when London came online. It should be obvious, but the results are very good and show just how important it is for us to be online in Europe.

From Amsterdam to OpenDNS before London goes online:
bash$ ping 208.67.222.222
64 bytes from 208.67.222.222: icmp_seq=0 ttl=57 time=145.077 ms
64 bytes from 208.67.222.222: icmp_seq=1 ttl=57 time=152.962 ms
From Amsterdam to OpenDNS after London goes online:
bash$ ping 208.67.222.222
64 bytes from 208.67.222.222: icmp_seq=0 ttl=58 time=9.814 ms
64 bytes from 208.67.222.222: icmp_seq=1 ttl=58 time=9.528 ms

The ping test above is a measurement of how long it takes one “packet” of Internet data to reach another host on the Internet. Bringing London online dramatically increases our reliability, speed and performance for our European users. Additionally, it decreases load in New York and Washington DC providing a win for our users in the US as well. Finally, the more sites we have, the more reliable our network becomes and that is a win for everyone.

We are online in London thanks to the efforts of a few really superb technologists and friends. I’d like to thank James Rice for his on-site help as well as his excellent guidance and advice along with Nick Waterman who fixed a minor issue we were having with our IBM BladeCenter chassis. Nick did this on December 29th, when he could have been at home with family or out partying; we really appreciate it Nick. James and Nick run Jump Networks, a high-quality, technically-savvy service provider in London.

I’d also like to give a big thanks to Chris Orme and Philip Baker from Datahop, a metropolitan fiber network in London which provides all kinds of really convenient network services. Chris worked tirelessly to make sure we could be online with our transit provider, NTT Europe in a timely fashion. It’s hard work getting folks to do things during the holidays and Chris made sure it happened. Even more impressive, Philip spent time late at night with our routers and switches making sure they were in good shape to turn up a BGP session with NTT Europe. James, Nick, Chris and Philip all went way above and beyond the call of duty and we appreciate it. I highly recommend both Jump and Datahop to anyone looking for transit and colo in London. Thanks guys!

We’re proud to be in London and look forward to peering with networks currently connected to LoNAP, a growing peering point in London. We are also considering a peering session at LINX, one of the largest exchange points in the world.

Happy New Year from everyone here at OpenDNS and we’ll see you in 2007!

My cousin was one of the many people to send me a link to David Pogue’s blog post at the New York Times titled “A Faster Web–for Free” about OpenDNS and how much he and his wife love it. That’s just awesome!

For those of you who don’t know, David Pogue is the consumer technology superhero for the non-geek population. In addition to his usual column he also does a bunch of really great video podcasts that you can watch.

Pogue writes in his column:

“In short, Open DNS works by caching Web-page requests from its thousands of users, so that the site you want is blasted to you in a fraction of a second.”

Just to clarify, we cache the DNS requests only. DNS is the part of a web request where your computer says “how do I get to site xyz.com?” DNS is a significant part of the latency in your web experience and that’s where a large part of the speedup comes from but we don’t actually cache web pages from other sites.

As of Dec 31, 2006, London is online.

On our network map, we show our four current network nodes in the United States, and provide insight into our future locations. The map, dated July 7, is still accurate as I type this.

The first location online from our “Coming soon” contingent will be London, England. Our hardware is racked and powered in the London facility. But we’ve been held up by bandwidth discussions, as we have some specific network requirements that complicate the matter beyond just the cost of connectivity.

The delay is frustrating to us, too. My apologies to the several folks who have inquired and been told (by me personally, or by my colleagues) that London would be online by this time. I’m not going to promise a new date right now, but we’re working on this, and will announce more details on our blog as we have them. Once the London location is online, we’ll focus more attention on our next locations.

Fortunately, many customers are finding that OpenDNS is faster for them in the UK already, despite any network latency. That’s proof positive that DNS speed is the combination of two factors: network latency and software speed/cache size. Even when we’re “farther” away on the network, OpenDNS often delivers results back to the end user faster. We want to accelerate the experience again, by removing the network latency concern — which is the whole point of London.

Is it only me, or does this post beg for The Clash’s London Calling? Or is that just too much of a cliché?

CNET just rehashed a report (pdf of report) that our friends over at Nominum commissioned to look at the speed and reliability of ISP DNS servers. The verdict won’t shock any of you: ISPs are pretty bad at providing DNS.

Some of the numbers they put in the report are surprising. The report says that Verizon drops 3.14% of all DSL subscribers’ DNS requests. That is some messed up DNS!

The report goes on to talk about other ISPs including SBC, RoadRunner, Comcast, who all do relatively poor jobs at providing such a critical service. I’m bummed they didn’t review Speakeasy, an ISP I’ve always really liked and whose DNS servers have always performed reasonably. The report states that Comcast only drops 0.51% of queries which is amusing because most people tend to attribute bad DNS service with Comcast. We know that the reliability of the DNS is important and we keep our system reliability statistics totally open and accessible. I challenge Verizon or Comcast to do the same.

What’s the point of all this? This report really shows that there is a lot of room for improvement in the DNS space and it clearly starts with reliability and performance, two things we cover well. Reporters still don’t understand the importance of DNS because it’s much more than just about speed. That’s one important part but the other is that DNS is a major part of the Internet and just like there are firewalls and anti-spam solutions, users needs tools to manage their DNS too.

Bringing this issue to light is a good thing. Even though we didn’t pay for or commission this report, I can’t help to think it was made to open ISPs’ eyes to our service. We’ve created an opportunity for ISPs where there was none before. OpenDNS provides these kinds of tools to users.

As much as I would love an ISP like Verizon to work with OpenDNS to make their users’ Internet better I would be upset if it was done arbitrarily and not on an opt-in basis. If I were a Verizon user currently using their three-percent-query-dropping DNS I’d switch to OpenDNS in a heartbeat. It’s easy to get started with OpenDNS right now.

A week ago, The New York Times published an entertaining article by John Markoff and Saul Hansell about Google’s new data centers in Oregon, “Hiding in Plain Sight, Google Seeks More Power.” Since the link soon may lead to TimesSelect (read: $), I’ll pull one sentence to show the larger point of the article:

Google, Microsoft and Yahoo are spending vast sums of capital to build out their computing capabilities to run both search engines and a variety of Web services that encompass e-mail, video and music downloads and online commerce.

Google’s reticence on the subject makes for some amusing anecdotes in the article, but mostly the article serves as a useful reminder that the Internet still obeys the laws of physics. Heat, energy, and physical space still matter, just in different ways.

Why do Google and others distribute their datacenters around the world?

Google has found that for search engines, every millisecond longer it takes to give users their results leads to lower satisfaction. So the speed of light ends up being a constraint, and the company wants to put significant processing power close to all of its users.

It’s not just search engines who need to deliver at (ahem) light speed. You can’t load google.com or yahoo.com or any other website without first making a DNS request (or several). That’s one reason (there are others, like redundancy & reliability) that OpenDNS runs its service from four geographically distributed locations, with more to come.

OpenDNS isn’t building datacenters, but we’re running our service from some of the best ones in the world. Also, we’re not so secretive that folks need to invoke Voldemort when referring to our company! From the article:

“No one says the ‘G’ word,” said Diane Sherwood, executive director of the Port of Klickitat, Wash., directly across the river from The Dalles, who is not bound by such agreements. “It’s a little bit like He-Who-Must-Not-Be-Named in Harry Potter.”

A note on being global

We know our coverage of the world beyond the United States can improve. London, England will be the next location online, probably in mid-July. Fortunately, in the short term, connectivity to the United States is quite good, and many Internet users outside the United States are relying on U.S.-based servers for much of their Internet experience already. That’s not ideal, of course. We want to be as fast for someone in Singapore as we are for someone in Seattle, but the speed of light will be a factor for now.

Let us know where in the world you are, as we make our future plans.