We take our operational excellence extremely seriously. We know that one of the reasons people choose OpenDNS is because they know they can count on us for reliable DNS resolution, something many ISPs can’t promise. And so when something happens that causes us not to be the reliable service we’ve promised to be, it’s a wake-up call for all of us at OpenDNS.

This morning, a major Internet provider had a serious routing issue in Southern California that caused some of their traffic to be lost before it reached our network. This type of failure didn’t trigger our monitoring system — because from our end, everything appeared online. Lasting approximately two hours in the early morning on the West Coast, the partial interruption in service was a localized one and did not affect the global OpenDNS service.

When issues like this happen we always like to step back and try to understand what we could do to prevent these kinds of problems from happening again. In the coming days we will determine how we can detect this type of problem more quickly and how we can respond more effectively.

To reiterate: we value our customers, and we take incidents like this exceptionally seriously. And while a disruption in service is never a good thing, we’ll take this as an opportunity to learn, and to ensure we run the most robust and globally available DNS service on the planet. I appreciate the years of support many of you have given us and hope you will continue to count on us to be the most reliable provider of DNS and security services on the planet.

DNS outages happen everywhere, from Italy to Illinois. And when they do, we can count on people taking to Twitter via their smart phones, to vent, find out what’s going on, and learn how they can get back online (thanks to us!).

We love helping frustrated people set up OpenDNS during these DNS outages. But one thing we discovered is that for the less-than-technical people amongst us, simply saying “use 208.67.222.222 & 208.67.220.220″ isn’t enough. People don’t know what those numbers are, or where to look on their computer or router to change them. That’s why we’ve created a new mini-site; you can find it at use.opendns.com or http://208.69.38.205/.

Why are we making it accessible via both an IP and a URL? Because if your DNS is down, we want you to be able to access the instructions via your computer’s browser. Having an IP address means no matter what’s happening with your DNS, you can get to the site.

The next time there’s a DNS outage, we’ll head to Twitter as we normally do, to act as a resource and problem solver for those without DNS. And, thanks to this new page, it will be easier than ever to get OpenDNS set up, even for those who’ve never heard the term DNS before.

But our hope is that we won’t be the only ones. Our hope is that you’ll bookmark http://208.69.38.205/ and that the next time you hear that there’s a DNS outage, you can be a resource for your friends. Text them, call them, tweet at them — let them know it’s easy to get back online and it’s simple to get safer, faster, smarter and more reliable Internet — all that’s needed is to set up OpenDNS.

Last night, millions of people across the eastern seaboard found themselves unable to go online and access the Internet. The culprit? A Comcast DNS outage that lasted more than three hours and affected customers from Boston to Baltimore. These kinds of attacks can hit anyone, including us. And it’s likely that if an attack was large enough to disrupt Comcast, it could be large enough to disrupt us. That’s scary.

When we launched introduced OpenDNS more than four years ago, our promise was this: the fastest, most reliable DNS service available. Since then we’ve added features and built out enhancements including malware protection, Web content filtering and SmartCache. That said, offering ultra-reliable DNS service is still at the core of what we do. This is part of the reason why we added a new datacenter in Singapore recently.

Over the past four+ years, we have been fortunate to have a perfect, 100 percent uptime record and we will work hard to maintain that. As we saw with Comcast last night, even great ISPs have outages when attacked with massive amounts of malicious traffic. This is why we will continue to add capacity, far in excess of what we actually need. The real solution to this is better security for end-users so they don’t get infected and become vehicles for DDoS attacks.

So, if you just got set up with OpenDNS last night, welcome! Hopefully last night was the last time you’ll ever be without Internet due to a DNS issue. We’ll work hard to make sure it was. If you’ve been set up with OpenDNS for a while now, you probably didn’t notice there was any issue at all.

The team here has been hard at work over the past few months getting a new datacenter set up, and I’m happy to report that as of today, our Singpore datacenter is online and serving production OpenDNS traffic. The Singapore server marks our 12th datacenter globally and the first of a number planned for Asia.

One of the benefits of OpenDNS is that we use a technique called Anycast routing in how we run our network. Anycast means that no matter where you are in the world, your DNS requests route through our closest datacenter. And when we do maintenance that requires us to take a site offline, our routing topology ensures you will route to our next closest datacenter. It also means that when we bring up a new datacenter that is closer to you, your DNS requests will automatically start routing to it. So for the bulk of our users in Singapore, Thailand, Vietnam, India and throughout Asia, this new Singapore datacenter promises an even faster Internet.

And here’s a photo of what our installation looks like as it was being racked:

This Singapore datacenter is only the start. We’re planning on adding a new datacenter in Frankfurt, Germany in early 2011. After that, the plan is to continue expanding our footprint from there. As always, you can take a look at our global system status on our Systems page.

PS — Internet routing is not a perfect science and requires a lot of work to get right. If you are in Asia and a traceroute doesn’t show you talking to Singapore for DNS we want to know! Please send a traceroute from your computer to 208.67.222.222 to our support department so we can see which networks in Asia aren’t seeing our new routes.

Yesterday, the DNS servers of Alice.it, one of Italy’s largest ISPs, went down for several hours. People who rely on Alice.it for their Internet waited. And waited. And then waited some more for their Internet to come back. Outages like these highlight the importance of a reliable DNS service; when it’s working, you don’t think much about it, but when it’s down, essentially so is your ability to access the Web.

As with similar outages in the past, like ones experienced in Germany and by Time Warner customers in 2008, users of OpenDNS didn’t notice any downtime at all. In fact, in all four plus years we’ve been around, OpenDNS has a perfect, 100 percent uptime record.

Because of the outage, we experienced a surge in traffic from Italy — not surprising, since word spread via Twitter and other methods that using OpenDNS was one of the only ways to regain access to the Web. We’d like to take the opportunity to welcome our new Italian users — benvenuto! We’re happy you’re here, and look forward to helping you enjoy a safer, faster, smarter and more reliable Internet.

This is my first blog post here at OpenDNS, though some of you may know me from answering your support emails. It is not unusual, however, for our own users to sit in the tech support seat themselves and offer help to others. Yesterday we received the following email, and I thought it was so good we should share it:

I’ve been studying abroad in the UK for three weeks and dealing with a crummy internet connection, but at least I had OpenDNS configured on my machine. This morning, the internet stopped working for everyone in the hotel… except me. It took me a bit to realize what had happened. Once I realized the Hotel’s ISP’s DNS was down, I spent the entire afternoon configuring OpenDNS on the laptops of everyone in the hotel lobby……we have to thank you for re-connecting us to the world, even across the pond!

Thanks for such an incredible service that anyone who understands networking would pay precious money for, yet you somehow provide it for free. It’s great to be immune to one of the most annoying internet problems you can have.

- Dominic

Thanks Dominic! While we are always happy to help, it is also nice when our users take the initiative to spread the word of OpenDNS and contribute positively to their community. After all, that is one of the ideas OpenDNS was founded upon.

(My thanks to Fabio Calvigioni (oRi0n) for translating this post into Italian, below.)

Update: Several commenters say the problem is gone, and that appears to be true. We hope it will continue. – July 23, 2007

We care about delivering a great Internet experience worldwide, and it appears that many Italians are poorly served by their current DNS infrastructure.

Several times over the last several months, Telecom Italia has blocked DNS requests from OpenDNS servers. What happens? Our customers in Italy are unable to resolve domains that uses Telecom Italia for their authoritative DNS, for example: pf.rossoalice.virgilio.it (a large Italian ISP).

For months, we’ve jumped through technical hoops to work around these blocks, even though all we’re doing is what any recursive DNS service would do: ask for the location of a domain from the respective authoritative server.

We’ve communicated through several channels: email, IRC, telephone. (We don’t speak Italian, but we’ve made an effort through translation services.)

Result? Nothing. (In DNS land, we’d call it a SERVFAIL.)

If anyone responsible and responsive at Telecom Italia wants to help us resolve this matter, we’re more than willing to work this out directly.

For all those Italian customers who have been affected now and again, we apologize. We are still trying to resolve this problem via normal channels. But we’ve been quite patient so far, and we’re a bit tired of banging our heads against this particular wall.

Here’s why we’re naming Telecom Italia.

First, dnsti.interbusiness.it and dnsts.interbusiness.it have blocked our requests.
Second, the interbusiness.it domain is owned by Telecom Italia, according to whois data.

Domain:             interbusiness.it
Status:             ACTIVE
Created:            1996-01-29 00:00:00
Last Update:        2007-01-30 00:36:13
Expire Date:        2008-01-29

Registrant
  Name:             Telecom Italia S.p.A.
  ContactID:        TELE616-ITNIC
  Address:          Via Paolo Di Dono, 44
                       Roma
                       00143
                       RM
                       IT
  Created:          2007-03-01 10:44:12
  Last Update:      2007-03-01 10:44:12

Nameservers
  dnsti.interbusiness.it
  dns.opb.interbusiness.it
  dns3.nic.it
  dnsts.interbusiness.it

I look forward to sharing a solution.

Telecom Italia non risponde ad OpenDNS

OpenDNS si impegna per fornire in tutto il mondo il miglior servizio agli utenti di Internet, ma sembra che molti italiani debbano fare i conti con un’infrastruttura DNS molto carente.

Per diverse volte negli ultimi mesi Telecom Italia ha bloccato richieste DNS dai server OpenDNS. Cosa succede quindi? Molti nostri clienti in Italia non riescono a risolvere nomi di domini che usano Telecom Italia come server DNS autoritativi. Per esempio, pf.rossoalice.virgilio.it (un importante ISP italiano) viene risolto in maniera errata.

Per mesi abbiamo utilizzato espedienti tecnici per aggirare questi blocchi, anche se tutto quello che stiamo facendo è quello che farebbe ogni servizio di DNS ricorsivo: richiedere l’indirizzo di un dominio al suo rispettivo server autoritativo.

Abbiamo provato a contattare Telecom Italia per email, IRC e telefono (non parliamo italiano ma abbiamo fatto dei tentativi utilizzando dei servizi di traduzione).

Risultato: nessuna risposta (nel campo dei DNS lo chiamiamo SERVFAIL).

Se qualche responsabile o addetto di Telecom Italia è interessato ad aiutarci a risolvere questo problema saremmo dispostissimi a collaborare direttamente in tale senso.

Chiediamo scusa a tutti i clienti italiani che hanno riscontrato e stanno ancora riscontrando questo problema. Stiamo ancora provando ad eliminare il blocco attraverso i normali canali. Fin qui siamo stati abbastanza pazienti ma ora siamo un po’ stanchi di doverci scervellare con questo problema.

Ecco perché crediamo che Telecom Italia sia la causa del problema.

Innanzitutto, dnsti.interbusiness.it e dnsts.interbusiness.it hanno bloccato le nostre richieste. Inoltre, il dominio interbusiness.it, secondo i dati del whois, appartiene a Telecom Italia.

Domain:             interbusiness.it
Status:             ACTIVE
Created:            1996-01-29 00:00:00
Last Update:        2007-01-30 00:36:13
Expire Date:        2008-01-29

Registrant
  Name:             Telecom Italia S.p.A.
  ContactID:        TELE616-ITNIC
  Address:          Via Paolo Di Dono, 44
                       Roma
                       00143
                       RM
                       IT
  Created:          2007-03-01 10:44:12
  Last Update:      2007-03-01 10:44:12

Nameservers
  dnsti.interbusiness.it
  dns.opb.interbusiness.it
  dns3.nic.it
  dnsts.interbusiness.it

Spero di aggiornarvi presto con una soluzione.

There’s no reason anyone should have to deal with flaky DNS. The students at Brown University went without Internet connectivity for 3 hours (!) yesterday because their DNS was less than reliable.

Brown University, if you can hear us: use OpenDNS!