We take our operational excellence extremely seriously. We know that one of the reasons people choose OpenDNS is because they know they can count on us for reliable DNS resolution, something many ISPs can’t promise. And so when something happens that causes us not to be the reliable service we’ve promised to be, it’s a wake-up call for all of us at OpenDNS.

This morning, a major Internet provider had a serious routing issue in Southern California that caused some of their traffic to be lost before it reached our network. This type of failure didn’t trigger our monitoring system — because from our end, everything appeared online. Lasting approximately two hours in the early morning on the West Coast, the partial interruption in service was a localized one and did not affect the global OpenDNS service.

When issues like this happen we always like to step back and try to understand what we could do to prevent these kinds of problems from happening again. In the coming days we will determine how we can detect this type of problem more quickly and how we can respond more effectively.

To reiterate: we value our customers, and we take incidents like this exceptionally seriously. And while a disruption in service is never a good thing, we’ll take this as an opportunity to learn, and to ensure we run the most robust and globally available DNS service on the planet. I appreciate the years of support many of you have given us and hope you will continue to count on us to be the most reliable provider of DNS and security services on the planet.

Yesterday, the DNS servers of Alice.it, one of Italy’s largest ISPs, went down for several hours. People who rely on Alice.it for their Internet waited. And waited. And then waited some more for their Internet to come back. Outages like these highlight the importance of a reliable DNS service; when it’s working, you don’t think much about it, but when it’s down, essentially so is your ability to access the Web.

As with similar outages in the past, like ones experienced in Germany and by Time Warner customers in 2008, users of OpenDNS didn’t notice any downtime at all. In fact, in all four plus years we’ve been around, OpenDNS has a perfect, 100 percent uptime record.

Because of the outage, we experienced a surge in traffic from Italy — not surprising, since word spread via Twitter and other methods that using OpenDNS was one of the only ways to regain access to the Web. We’d like to take the opportunity to welcome our new Italian users — benvenuto! We’re happy you’re here, and look forward to helping you enjoy a safer, faster, smarter and more reliable Internet.

Yesterday, OpenDNS resolved its 200,000,000,000th request. That’s 200 billion with a b.

Thank you to our many customers for making OpenDNS the world’s largest and fastest-growing DNS service. It’s been less than 3 months since we responded to the 100 billionth request. We’ve averaged more than 1 billion requests/day for that period. On Thursday, OpenDNS responded to 1.4 billion DNS requests (stats).

All that, with zero downtime (system status), thanks to multiple servers spread across five geographically distributed locations connected by multiple Tier 1 telco providers to deliver an amazingly fast, reliable Internet experience no matter the size of your network.

Come on in! Get started.

The magic numbers:

• 208.67.222.222
• 208.67.220.220

Yesterday marked another milestone in OpenDNS’s history; we served our 100 billionth DNS request. I remember how excited we were to serve 1 billion DNS requests in our first month. Compare that to yesterday where we served 1,071,072,782 queries in a single day.

This milestone is also a good time to point out that our DNS service has had zero downtime since the day it launched. We put up a system status website a couple weeks after we launched to show you just how reliable we are.

Or put simply…

And growing…

P.S. Some of you have asked what the 100 billionth DNS request was. Since we have five locations and we process stats in large aggregates every few minutes it is hard to tell. If we had to guess, we’re pretty sure it was either for TechCrunch or Digg.

I’ve just posted about this on the OpenDNS System Status site, but the OpenDNS.com website (and blog.opendns.com, etc.) were unavailable due to authoritative DNS failure for about 90 minutes earlier today, starting around 1pm PT (21:00 UTC). Here are the details.

I will repeat myself on a few key points.

First, OpenDNS’s speedy, reliable DNS was not affected. Our website is treated separately from our DNS, for this reason among others.

Second, the cause of the failure was a Denial of Service (DOS) attack on EveryDNS. David Ulevitch, CEO of OpenDNS, also owns and operates EveryDNS, but the two companies are separate. OpenDNS has used EveryDNS services, although we’ve now spread the authoritative DNS for OpenDNS more broadly as a result of this incident.

Third, the DOS attack on EveryDNS continues. It’s being actively worked on, as you can imagine. As we learn more, we’ll share it in this post, since I know other EveryDNS customers are interested, too.

Update: As of 9:30pm PT, December 1 (05:30 UTC December 2), EveryDNS is recovering. Still under attack, but mitigated. Status report on the EveryDNS home page. I’ll leave it to EveryDNS for updates from here on.

Note: you can bookmark or save our OpenDNS System Status site at http://208.67.219.60/ just for rare events like this, whether there is an authoritative or recursive DNS issue.

Yesterday we crossed the 4,000,000,000 (That’s four billion!) overall DNS query mark.

More importantly, we also crossed the 100,000,000 (That’s one hundred million!) DNS queries per day mark.

Check it out:

Woo Hoo!

As of Dec 31, 2006, London is online.

On our network map, we show our four current network nodes in the United States, and provide insight into our future locations. The map, dated July 7, is still accurate as I type this.

The first location online from our “Coming soon” contingent will be London, England. Our hardware is racked and powered in the London facility. But we’ve been held up by bandwidth discussions, as we have some specific network requirements that complicate the matter beyond just the cost of connectivity.

The delay is frustrating to us, too. My apologies to the several folks who have inquired and been told (by me personally, or by my colleagues) that London would be online by this time. I’m not going to promise a new date right now, but we’re working on this, and will announce more details on our blog as we have them. Once the London location is online, we’ll focus more attention on our next locations.

Fortunately, many customers are finding that OpenDNS is faster for them in the UK already, despite any network latency. That’s proof positive that DNS speed is the combination of two factors: network latency and software speed/cache size. Even when we’re “farther” away on the network, OpenDNS often delivers results back to the end user faster. We want to accelerate the experience again, by removing the network latency concern — which is the whole point of London.

Is it only me, or does this post beg for The Clash’s London Calling? Or is that just too much of a cliché?

If any of you look at our stats page as often as we do you might have noticed that the weekly graph was a bit wonky today and that we (temporarily) dropped the daily graph. I put a note up on the stats page explaining it but I’m putting a note up here for posterity.

Basically, in order to keep DNS as reliable and awesome as possible we process all of our DNS stats on other machines, out of the DNS traffic’s line-of-fire to you. The machines we use right now to do stats needed some more RAM and hard drives so we paused the stats processing, had some lunch, added some disks and RAM, and then went home. Sometime this morning we decided to start up stats processing again and now it’s nearly all caught up. So there ya have it, the stats were kinda wonky, but our DNS service was hummin’ along like the well oiled machine it should be and is.

As for future changes to the stats page, I have a lot of ideas in my head but I want to know what you want. What kinds of stats are important to you? I want to be sure what’s important to me isn’t just important to me.

On on,
David Ulevitch

Update: I just got teased by some of the guys here for ‘signing’ a blog post. Well now there’s a smiley too.