Packet 8 Suffers DNS Related Outage

Interesting, I've always thought of the registrars as a pretty critical single point of failure. Does anyone have a creative solution for this?

Not a very good one. You could have a primary / secondary setup where you might have sipserver.primarydomain.com and sipserver.secondarydomain.net and have your domains registered at two different places. Your equipment would have to support this and you would have to use DNS servers with different domains with different registrars.

That's all I've been able to come up with also. I was thinking more related to web infrastructure, so in my case we'd have to somehow notify our clients to use a different FQDN. Not exactly ideal or really even feasible at all.

Stagger the renewal dates.

For a single domain? There's only one date.

How about, don't dick around with DNS settings when you're doing a renewal? If there's one thing I've learned about DNS, it's "one thing at at time". If you're doing a transfer, don't switch hosts for a week or so. If you're moving to a new host, don't try to transfer registrars.

Static IP with DNS backup.

Basically, the device uses the same static IP and if the login for it doesn't work, check the DNS to refresh the new static IP. Check backup DNS if the first one fails for login. Pretty straight forward really, that's what my Vonage router does, it can work in a DNS outage.

That's a good idea, but the problem with it is that the device has to support it. That doesn't translate well to a web infrastructure, or any email where the user has to input the information to reach the site.

I was thinking of something along the lines of multiple registrars for a single domain ... and if one returned an NXDOMAIN or any other sort of error, DNS would check the secondary registrar. I think my idea would require a significant change to the entire DNS infrastructure though.

Sue them for loss of business so next time they won't screw up. It's not like this has never happened to anyone else out there...

I agree with the above. There are multiple solutions to preventing this problem. A company like Packet 8 should not be blaming the registrar and should be taking measures to make sure it does not happen again.

You should have different domains at different registrars and stagger the renewal dates. I would even count on a third backup option in the config of simply using an IP address in case the clients DNS servers failed, you could still provide service to the client.

So you would list:
SIPSERVER1.DOMAIN1.COM
SIPSERVER2.DOMAIN2.NET
63.209.12.1

VOIP really needs to get up and beyond pots. As Verizon and others are coming "down" to VOIP type solutions other VOIP providers also need to "come up" to meet those reliability standards. For a change, VOIP is started to be put on the same level as major carriers, they just need to have the same reliability, services, and pricing.

Agreed, I think packet8 is trying to pass the blame. I think someone made a booboo on the dns renewal page and the system wasn't designed to be robust enough to handle such a failure.

Packet8 just emailed the following explanation and resolution procedure for correcting the Register.com DNS failure:

At 7 pm last night (January 22, 2008) Register.com changed the DNS for »www.packet8.net and »www.packet8.com by omitting the DNS and substituting a landing page in its place. The net result was call failures and the inability to reach Packet8's website. Our telephones and DTA’s have several hard coded fail-over processes built into them. However, due to the landing page put up by Register.com, the end points were given a false signal of success and did not fail over to the backup IP addresses built into each device.

Within minutes Packet8 engineers saw the issue, contacted Register.com and got the issue resolved with proper routing instructions broadcast to all DNS servers on the Internet.

Most Internet Service Providers updated to the correct DNS routing instantly. However, we have reports that ATT, ATT-Mobile and Time Warner Roadrunner on the East coast have not updated DNS servers with the correct information.

If you are having issues with the Packet8 service or reaching our self-service portal, please provide the solutions below to renew the DNS information in your modem, router, and computers.

If the solutions do not work, your ISP may be providing outdated DNS information. Please call Packet8 support with the following information and we will contact your ISP regarding the issue.

Call Packet8 Support at 1-888-898-8733 or, if outside the US, call 1-408-687-4120

Solutions:

1. Point the DNS server settings of your Packet8 endpoints and telephones to 63.209.12.18 or set your router’s DNS settings to Open DNS with 208.67.222.222 and 208.67.220.220.

2. Edit your hosts file to force »www.packet8.net to 63.209.12.100

3. Provide a network power cycle with step by step instructions shown below

Network Power Cycle:

Power cycling the entire network refreshes and re-syncs all network devices with the most current network information broadcast from the ISP.

1. Unplug power from the back of all network devices (modem, router, & Packet8 device) & shutdown any computers. Then wait one minute.

2. Plug the power cord back into the modem and wait one minute to let the modem synchronize with the ISP. (Check for ONLINE/Internet light)

3. Plug the power cord back into the Router and wait one minute.

4. Plug the power cord back into the Packet8 device and wait 30 seconds

5. Check the Packet8 phone for a dial tone. Also, the PHONE LED on the Packet8 device should come light up when the receiver is picked up or turned on.

6. Check lights: solid POWER LED & an occasionally flickering LINK LED

If no dial tone, turn on a computer and make sure the customer can browse the internet.

Clear computer of old DNS information:

(Windows) Start --> Run --> type in: cmd --> type in: ipconfig /flushdns
(Mac 10.4-) command: lookupd -flushcache
(Mac 10.5+) command: dscacheutil -flushcache

awesome, thanks...best response yet!

And that is nevertheless your failure for failing to a.) entrust register.com with servicing all of your domains, and b.) designing a proper, SECURE service mechanism: your software is unable to tell that the server it is talking to is not the "real you"? Ever heard of SSL/TLS and signed certificates?

So in the current state, every botnet/trojan that changes a Windows hosts file (which you yourself advocated as a temporary workaround) with a www.packet8.com entry (or certain others controlling service for your client) can disrupt service for your endusers, or worse, possibly redirect the SIP clients to wherever the attackers want them to point to? I sure hope for everyone's sake that that condition does NOT apply here.That is the most ridiculous, uneducated, mis-leading and patently false description of how DNS supposedly works I have ever heard. You honestly think you can make people believe that you can broadcast information to every single one of the millions of nameservers on the Internet? And that your problem is just that a lot of them are not listening to you? We wonder why!

There is no "broadcast". There's only "pull" - from client resolvers to authoritative servers of your domains. Too bad the DNS records register.com's DNS servers were dispensing for that period were the wrong ones - there is no possible recall of that information during the TTL (time-to-live) period that was attached to those bad records THE MOMENT THEY WERE SERVED.This second statement, once again, is uneducated, misleading, patently false and of defamatory character: there's nothing these ISPs are doing wrong, and there is no action required on their part. Period. They got served (only obvious to you: bogus) DNS records with a set TTL (time-to-live) by register.com for your domains, and they are PROPERLY SERVING THESE RECORDS to their subscribers for the ENTIRE DURATION OF THAT TTL. And only after that TTL has expired, will they re-resolve these names/records, if one of their subscribers requests DNS resolution for them. You got a problem with the TTL and what register.com dished out on your behalf, you say?

I have no doubt in my mind, and firmly believe that these falsehoods are intentional on Packet8's part: throw up a smoke screen, blame register.com, blame the ISPs, blame everyone else but yourself for this shooting yourself in the foot bigtime - starting with a single point of failure. You probably regret entrusting register.com with your service by now, as it was (from the description) register.com that temporarily suspended service to the domains, but started to serve bogus DNS records for both the nameservers (NS records) and hostnames (A records) for the domains (with a TTL of unknown duration):

register.com permitted themselves to do this in the fine print of your contract with them maybe, but it was nevertheless service-disrupting in a big way for you: learn how to read the fine print already: you could have picked a registrar that doesn't try to generate revenue by leeching click-through traffic by putting up their own NS/A records in place of your own the moment a domain is suspended or expires: a patently dishonest "monetization scheme" in my humble opinion. Now imagine what happens if a large DNSBL's domain is suddenly shut down this way...and the world's email begins to bounce (a lot of it), as every DNS request is answered with an IP number (of the landing page's server), indicating an active blocklisting condition (and don't get me started on replies other than 127.x.x.x that should not be treated as such)....And you expect ISPs big and small to clear their collective 1000's of recursive caching DNS servers JUST for you, after this ridiculous finger-pointing and blame-game you're trying to play here? Cache invalidations/reloads in the middle of the business day, that are impairing server performance for millions of users and millions of other domains that are working fine?

I can tell you exactly where you'd get with this over here: we'd laugh hysterically until we'd hang up on you - followed by us instructing customer service what to tell complaining customers: that we are aware that packet8 has a DNS problem of their own making, which will probably resolve itself in "the next 24 hours" - and to feel free to call back tomorrow, if the problem hasn't resolved itself. I have a good idea who they're going to call next.

These guys are losers plain and simple. Their JV, to say the least. Look at the CEO and all his garbage with the financials every quarter. Then, take a look at the highly incompetent Board of Directors (all of which should have been retired 30 or so years ago).

Add all these up and what you have are a bunch of wanna-be executives who want to run your phone service.

Buyer beware. These guys are clowns.

Good Job Anonymous Coward, you went straight to the heart of the issue. Obviously Packet 8 hates their customers with a passion. Why else would they bother to post on this website the troubleshooting fix necessary to restore service. Nothing about this outage points to a vicious scam against anyone yet you seem eager to jump on very circumstantial evidence in order to make an isolated incident seem much worse than it is.

The nature of the service Packet 8 and other VOIP companies provide does make them susceptible to general networking and internet issues. This does not mean though that these companies are indifferent when an issue like this occurs. VOIP users need to understand that these issues will sometimes be inevitable but any good service based company will be able to work through most issues quickly and efficiently.

Register.com and Network Solutions are responsible for more DNS screw up than all other registrars combined IMHO.

Godaddy rocks. I have been using godaddy as my registrar for almost 10 years without one hiccup.

If you use register.com or network solutions you get what you deserve.

It really bugs me when people pass the buck. The reason they went down was because someone from their company logged into their account at Register.com and changed the DNS. I'm in a position to know. Sue, go right ahead, but they won't because every change is documented and who made the change. They should own up to what they did. If anyone should be doing the sueing it's Register.com. The statement is false and slanderous. I sure Register.com's lawyers will have a field day with this.

Let me pick a part a lot of what your saying:
1.You say Packet8 failed by entrusting register.com, but you fail to mention that a lot of major companies, such as Cox, Verizon, AT&T, Brighthouse, ect. Where they did fail is by NOT faulting over to a different DNS
2. Yes, with a domain you have a seperate DNS server, which is then redirected to each one of the Packet8 users. How can a company be prepared when their domain host changes the server name and location on them without notification? If you owned a website being hosted by a smaller domain holder, and they made the change on you, how would you cope?
3. NO ACTION REQUIRED BY ISP'S? Are you crazy? They, just like any other adverage Joe, MUST reset their DNS tables after an outage, and they did. They are quicker to the punch than packet8, because they have to be. They are an extremely large company with fault over servers, packet8 is not
4. Have you even checked your sources other than with Packet8, because register.com has released statements of appology, Verizon has notified that their customers should all be back up, although some might not be and need a dns flush. So where exactly does the smokescreen exist?
5. No, I dont expect ISP's to refresh it just for packet8, I expect them to refresh for my sake, the dsl consumers! As i said before, this is not only a fault of packet8, but of a lot of other companies as well
Finally, I dont not like packet8 anymore than the other people here, you can even say i loathe it, but there is no way that you should place the blame solely on them. There does need to be blame on them, yes, but not all of it. there is a lot of companies the "blame game" should be divided between, one being Register.com

It would be nice if they would have at least put up a recording. Only communication I received was a Packet 8 recorded message when calling customer service, "All lines are busy please call back later". Sent emails but those went unanswered as well. Went most of the day without any phone service.

Disgusted and mad

And yet even more high fives are passed around at the OpenDNS headquarters.