smcallah
join:2004-08-05
Home
| reply to CapinPete
Re: That's right
But it was Level(3) that turned this back up because THEIR customers were complaining to them.
Do you think that having so many large dialup ISP's and RoadRunner as their customers that they weren't catching any flack from this? They did this to appease their own customers. There is no reason to think otherwise. Level(3)'s customers were complaining to Level(3), and Level(3) stopped it.
If Cogent lets it drop on Nov. 9, then Level(3) can have a better excuse to give their customers. But I doubt it will stop them from complaining. |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| said by smcallah  :If Cogent lets it drop on Nov. 9, then Level(3) can have a better excuse to give their customers. I think with it out that Level(3) gave Cogent over 75 days notice, they already have a good excuse... Cogent dropping the ball on Nov. 9th would just be a repeat of what happened the other day.
--
Tor server operator... Helping the free flow of information daily. |
|
smcallah
join:2004-08-05
Home
| Level(3) can't use the 75 day notice, if it does indeed exist, as an excuse. Because they didn't notify their customers at all before they turned it off. Not 75 days before, not 30 days before, not 1 week before, not 24 hours before. Not even 1 hour before.
The Nov. 9th date is in an official press release. |
|
jnxrox
join:2005-10-07
Las Vegas, NV
| Take a step back
Level3's general customer profile is not a small/med, single-homed business. They cater to large multi-homed carriers and businesses. Cogent's customer base is much more vunerable here. It's the single-homed level3 customer to the Cogent single-homed customer that is impacted and as low cost vendor to the masses, that fits the cogent profile more. Level3 isn't going to connect to transit to satisfy such a small % of customers just to get to Cogent's single-homed customers. Cogent has more skin in the game and you saw that desperation in their initial responses.
Disconnecting peering happens. Cogent was notified and did not make arrangements for their customers. Level3 disconnected and that was their plan. If you check the reality on the connectivity affected, it's smaller than Cogent would lead you to believe. Most companies are multi-homed and in fine shape for re-routing around these issues. |
|
BosstonesOwn
join:2002-12-15
Everett, MA
clubs:
 ·Comcast
| reply to smcallah
Re: That's right
said by smcallah :But it was Level(3) that turned this back up because THEIR customers were complaining to them. Do you think that having so many large dialup ISP's and RoadRunner as their customers that they weren't catching any flack from this? They did this to appease their own customers. There is no reason to think otherwise. Level(3)'s customers were complaining to Level(3), and Level(3) stopped it. If Cogent lets it drop on Nov. 9, then Level(3) can have a better excuse to give their customers. But I doubt it will stop them from complaining. Level 3 customers (yeah I use them) never had a problem reaching cogent or at least none of my servers. The had another peering agreement in place to take up the slack of the cogent drop. GCW ? I think it was. all be it it took a couple more hops to hit SJ but it still got there in under 200 ms. So I am happy.
--
"It's always funny until someone gets hurt......and then it's absolutely friggin' hysterical!" |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| reply to smcallah
said by smcallah :Level(3) can't use the 75 day notice, if it does indeed exist, as an excuse. Actually they can. By giving Cogent at least 75 days notice, it gave Cogent the oppurtunity to prepare for the peering sessions to be ended.
Because they didn't notify their customers at all before they turned it off. Not 75 days before, not 30 days before, not 1 week before, not 24 hours before. Not even 1 hour before. They don't have to nor should they have to, otherwise Level3 would be required to notify customers of every tiny change. All they are required to do is tell Cogent. Once Cogent is notified, the ball is 100% in Cogent's court - they either fix their traffic profiles or they take measures to ensure routes to Level3.
As it stands, from the information available, Cogent totally dropped the ball here and this problem, considering they were given notice, appears to be their fault for the most part. They knew they would be cut off and failed to ensure they had alternate routes to Level3.
--
Tor server operator... Helping the free flow of information daily. |
|
smcallah
join:2004-08-05
Home
| said by bmn :They don't have to nor should they have to, otherwise Level3 would be required to notify customers of every tiny change. Tiny change? You think THIS was a tiny change?
Level(3) DOES notify its customers of major changes in the network. Most large providers I know do this, it's in their contracts with their customers. Especially ones with Service Level Agreements that require daily or monthly credits to be given when that SLA is not met.
I can't believe you consider this a tiny change that Level(3) made. |
|
smcallah
join:2004-08-05
Home
| reply to BosstonesOwn
Level(3) did not have any Cogent routes on their network since Wednesday, that was easy to see with their looking glass server. They had no alternate routes to Cogent, or this wouldn't have been a problem.
RoadRunner, one of Level(3)'s largest customers, certainly couldn't reach Cogent through Level(3), as evidenced by all the posts from RoadRunner customers here. |
|
smcallah
join:2004-08-05
Home
| reply to jnxrox
Re: Take a step back
Well, since Level(3) turned this back on because of customer complaints, obviously there was a lot of need for their customers to get to Cogent.
If it was as "small" an impact as you say, Level(3) wouldn't have been bothered by a "few" customer complaints.
Also, sometimes multihomed customers don't want to use their backup connection, because they may be paying more per megabit for it. Say you're paying Level(3) for a 155 mbps connection, and it's $50/mbit. And you have another connection to Provider X, who charges you $150/mbit, but ONLY if you use them. So that peer is running at 0, you're paying them nothing until you use it. Then suddenly 20% of your 155 mbps goes to your backup connection. All of a sudden you're looking at paying an extra $4600 a month that you didn't have to pay, because now one of your providers doesn't have any routes to Cogent.
Many multihomed customers are setup like that. They have a main link, and a backup. Sure, it was their choice to get such a link. But it is people like this that can pressure Level(3) to do what they had to do. I worked for a year for a small hosting company that had a Level(3) connection and a connection to another ISP that worked like this. |
|
vferrari3
join:2005-10-07
Boston, MA
| said by smcallah :Well, since Level(3) turned this back on because of customer complaints, obviously there was a lot of need for their customers to get to Cogent. If it was as "small" an impact as you say, Level(3) wouldn't have been bothered by a "few" customer complaints. It wasn't turned back on because of customer complaints. It was turned on to help Cogent which was the more desperate party here. Notice the lack of information from Level3 as opposed to the screaming and press releases coming out of Cogent. It will be the same in Nov., Level3 will not pay to reach a small % of single homed cogent customers.
Mostly you'll still use the Level3 link to get to the many Cogent multi-homed customers, you'll just take a different path. But, yes in your scenario you would start using the unused link to get to the single-homed Cogent customers. Tell me why this is Level3's problem? They run their business, you run yours. With all the people up in arms there's one thing that seems to always be forgotten...this is not an unusual thing to happen, the Cogent response is what is the unusual piece. They did the same thing with others in the past and had to purchase routes to fill in their 'gaps'. You never hear about the other times carriers de-peer with each other because it's taken care of during the period of notice. Cogent uses the period to call the other carrier's bluff, and when connectivity is lost they smear. It's happened more than once.
This was more a PR war then anything and Cogent won the battle.
_VF |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| reply to smcallah
Re: That's right
said by smcallah :said by bmn :They don't have to nor should they have to, otherwise Level3 would be required to notify customers of every tiny change. Tiny change? You think THIS was a tiny change? No, but you misunderstood, so I'll clarify...
Level(3) shouldn't need to be required to notify its customers of a peering change, otherwise then they would have to start to notifying their customers to tiny changes to the network. Is that more clear ?
Level(3) DOES notify its customers of major changes in the network. Most large providers I know do this, it's in their contracts with their customers. Especially ones with Service Level Agreements that require daily or monthly credits to be given when that SLA is not met.
I can't believe you consider this a tiny change that Level(3) made. Actually, you are right, this isn't a tiny change in the network, but I never considered it one. The point here is that it was Cogent's responsibility to make sure that it had routes to Level3 once Level3 notified them about the peering sessions being dropped. Level3 incorrectly assumed that Cogent would do ITS job and find alternate paths to their network via other transits providers.
--
Tor server operator... Helping the free flow of information daily. |
|
smcallah
join:2004-08-05
Home
| I still think you misunderstood.
Obviously Level(3) customers were pulling a lot of traffic from Cogent, or there wouldn't have been an issue here.
So this peering change would have been a change worthy of notifying their customers. They didn't notify their customers of it. And suddenly their customers' customers were complaining, and Level(3)'s customers could only say, "Not our fault." But that doesn't make customers understand the problem, even if it was explained to them, a non-technical customer isn't going to understand, neither are the ones that think they know what BGP is.
So Level(3) had RoadRunner and several large dialup ISP's fielding calls, giving credits, and losing customers during the almost 3 days that this was down.
I don't see how you think that notifying their customers of a major peering change would suddenly open them up to having to notify of every tiny change.
And what makes it ONLY Cogent's responsibility to make sure they could get Level(3) routes? Why wouldn't Level(3) have a resposibility to give their own customers routes to Cogent? And now obviously they have taken up that responsibility, by turning this back on because of customer complaints. |
|
smcallah
join:2004-08-05
Home
| reply to vferrari3
Re: Take a step back
said by vferrari3 : It wasn't turned back on because of customer complaints. Sorry, but it was because of Level(3) customer complaints that this was turned back on.
Level(3) didn't care about helping Cogent, they cared about helping themselves. If they cared about helping Cogent, they would have extended this without turning it off.
But since Level(3) also had losing customers and losing revenue on its mind, they had to turn this back on. They were the only ones with the ability to turn it on.
It doesn't matter if it's Level(3)'s fault or not. The only reason they exist is for customers. If they lose those, what do they have? |
|
bmn
? ? ?
Premium,ExMod 2003-06
join:2001-03-15
hiatus
| reply to smcallah
Re: That's right
said by smcallah :I still think you misunderstood. Obviously Level(3) customers were pulling a lot of traffic from Cogent, or there wouldn't have been an issue here. No, the issue is that Cogent doesn't have enough users on its network to balance out the amount of traffic being sent to other networks. I recommend checking out when AOL dropped its SFI with Cogent and when C&W dropped its SFI with Cogent
said by smcallah :And what makes it ONLY Cogent's responsibility to make sure they could get Level(3) routes? Why wouldn't Level(3) have a resposibility to give their own customers routes to Cogent? And now obviously they have taken up that responsibility, by turning this back on because of customer complaints. Because, as an upstream Tier 1, Level3 receives the routes TO Congent's network FROM Cogent or via Cogent's transit providers. Without receiving those routes from Cogent, Level3 has no way of knowing how to get there... Level3 can't magically guess how to get to Congent without getting BGP information from Cogent. Level3 is not the one responsible to make sure that it can even see Cogent's networks since they don't have any control over Cogent's BGP information.
And the reason that Level3 took up the responsibility of getting routes to Cogent is because Cogent did not.
--
Tor server operator... Helping the free flow of information daily. |
|
vferrari3
join:2005-10-07
Boston, MA
| reply to smcallah
Re: Take a step back
said by smcallah :Sorry, but it was because of Level(3) customer complaints that this was turned back on. Level(3) didn't care about helping Cogent, they cared about helping themselves. If they cared about helping Cogent, they would have extended this without turning it off. You don't get it. They have no obligation to extend without turning it off. Peering is for mutual benefit. When one party no longer perceives benefit they back out. The party that relies on that arrangement too much is the weaker of the two. Level3 turned it back on because clearly there were PR implications where the public was not understanding the issues and they gave Cogent a reprieve. They knew what they were doing, they've done it before and come november, they will still not have those Cogent routes unless cogent utilizes their transit providers. Level3 did a risk analysis and/or cost analysis. The arrangement doesn't work for them. There's nothing wrong with that. They believe that the cost to keep the peering up is too much and is not balanced by the revenue they take in from the smaller single homed customer that were affected. If those customers go away, it may even be that they perceive that as a good thing. Smaller customers in the wholesale carrier business are usually higher cost to operate and higher cost to provision. It's a business decision and if you don't like it, take your business elsewhere. Again, I applaud that decision and there's nothing wrong with that. Forcing Level3 to keep up links that are not beneficial to their business is what would be wrong.
How would you like it if you were forced to keep a business relationship going that you perceived as bad for your business? You'd want out if an arrangement with a supplier was no longer benefitting you.
_VF |
|
vferrari3
join:2005-10-07
Boston, MA
| reply to smcallah
Re: That's right
said by smcallah :I still think you misunderstood. So Level(3) had RoadRunner and several large dialup ISP's fielding calls, giving credits, and losing customers during the almost 3 days that this was down. I don't see how you think that notifying their customers of a major peering change would suddenly open them up to having to notify of every tiny change. And what makes it ONLY Cogent's responsibility to make sure they could get Level(3) routes? Why wouldn't Level(3) have a resposibility to give their own customers routes to Cogent? And now obviously they have taken up that responsibility, by turning this back on because of customer complaints. That's what the notice was for. Level3 is in the position of strength. They gave Cogent the notice to make arrangements to get to Level3 after the peering goes down. Level3 has the eyeballs and that's the position of strength. I know you don't like it, but this is the way it is and has been. Cogent is the one with the history of not being able to play nice and has been through this before. Most carriers respect the model and when peers de-peer them, they make the arrangements.
BTW, what does RR have to do with this. RR should be multi-homed and should not be caught up in this. If they were, shame on them. I think you're over estimating the number and type of customers that Level3 would be losing. Again, for their business model they may even expect to lose the lower end of the base in order to continue mining the higher end of their base. They have multiple 10Gb links to some customers, the single homed DS3 customer leaving is not affecting them.
_VF |
|
DSLTech
join:2000-12-30
San Jose, CA
| reply to vferrari3
Re: Take a step back
I know for sure Level 3's customers were complaining. Can you imagine what a handfull of unhappy ISPs complaining might do to convince Level 3 to make a change?
Who was complaining? Oh, perhaps ISPs like Earthlink, Speakeasy, Megapath, etc. who all have connections via Level 3 to the internet, as well as other forms of peering. Dont you think that their complaints weigh heavily on Level 3?
Earthlink could, for example, choose to use Wiltel instead of Level 3 as a primary internet access connection, so their displeasure is very, very important to Level 3.
Fortunately our ISP is dual-homed to the internet, Level 3 being only one of the 2 options. |
|
smcallah
join:2004-08-05
Home
| reply to bmn
Re: That's right
said by bmn : Level3 is not the one responsible to make sure that it can even see Cogent's networks since they don't have any control over Cogent's BGP information. Obviously Level(3) has TOTAL CONTROL over the BGP routes it sees from Cogent on its network.
They can peer with Cogent, if they drop this peer, they lose the routes. They can also peer with a transit provider of Cogent, if Cogent has such a transit provider, and receive those routes. But Level(3) can also CHOOSE to drop those BGP routes.
Why do you think Level(3) has no control over Cogent's BGP routes on its network? |
|
smcallah
join:2004-08-05
Home
| reply to vferrari3
Re: Take a step back
I don't get it? My whole point was theoretical. Obviously it wasn't the true case. I was saying IF Level(3) cared about Cogent, they would have tried to work something out with them or extend the deadline without turning it off. They did not. So they spent 3 days getting customer complaints, which they DO care about. And so had to end up extending the deadline anyway.
But they obviously didn't care enough about their customers to let them know beforehand, so maybe my point is moot anyway.
And if the links aren't beneficial to Level(3)'s business, why did they turn them back up in the face of customer complaints?
It was customer complaints that brought this back up, whether you know that or not. What else would have brought it back up? Level(3) obviously wasn't willing to work with Cogent, for whatever reason, and dropped the connection.
They're more willing to work with them now, in the face of customer complaints, I hope anyway that it means they're more willing. It may simply mean that they are telling their customers that in 30 days you won't reach Cogent through them, and they'll end it at that with no negotiation. |
|
smcallah
join:2004-08-05
Home
| reply to vferrari3
Re: That's right
said by vferrari3 :BTW, what does RR have to do with this. RR should be multi-homed and should not be caught up in this. If they were, shame on them. I think you're over estimating the number and type of customers that Level3 would be losing. Again, for their business model they may even expect to lose the lower end of the base in order to continue mining the higher end of their base. They have multiple 10Gb links to some customers, the single homed DS3 customer leaving is not affecting them. _VF Why are you even asking what does RR have to do with this? RR is a huge cable ISP. They use Level(3), they might be Level(3)'s largest customer. You don't think RR was complaining? And RR is multihomed, they appear to use Level(3) and ATDN in most markets. But the fact remains, that they complained to Level(3) as well. So THAT'S what they have to do with this.
You're definitely underestimating the customer complaints that would have been coming into Level(3). Especially since they were the reason that Level(3) had to turn this back up. Now that they've given their customers a 30 day notice, perhaps it won't be so bad for them on November 9th.
If Level(3) didn't stand to lose revenue or customers, they wouldn't have had to make a move. How do you even doubt this? Have you seen their financials? They're going to take a beating this month alone on service credits they had to give to keep customers from wanting to leave.
I for one wouldn't want to be fielding support calls for something I couldn't support. Which is what RR, some DSL, and some dialup ISP's hanging off of Level(3) had to do. So they passed that pain on to Level(3). At that point, Level(3) is responsible to its customers. Sure Cogent is responsible to its own customers on the other end. But Level(3) had the ability to immediately get their own customers fixed, and they did this, 2 1/2 days later, without a notice before the cut off.
If you can't see that from the Level(3) customer and the customers' customers standpoint, you need to drop out of this discussion, because you clearly do not get it. |
|
