TomS_
debugger it
Premium,MVM
join:2002-07-19
Australia
1 edit | Staging a new network
All of the gear | 
Mess of power cables |  |
21 Cisco devices, mostly 2811 routers, all 3560 series switches, and a 7401 hand-me-down from elsewhere in the network that was no longer necessary.
Some gear has already been rolled out so is not pictured.
4-5 new towers are also being built. Most of this gear will be located at those towers, with some being located at a PoP elsewhere, and some colocated at an existing tower.
I'll be trying to get photos of the rest of the gear to share as we roll it out.
The antennas sticking out of some of the 2811's are for an Out of Band remote access solution (using HWIC-3G's) which is used when the primary network goes down so that we can get to things like console servers, masterswitches, etc.
Enjoy.  |
|
kewlkeed
Grouch
Premium
join:2005-02-05
Knowlton, QC | Awesome! I love benching a huge network on my desk hehehe.
I just did one recently with 25 nodes for what will be spread across nearly 500KM, all running gigabit speeds. I never took pics sadly... I'll remember for next time. |
|
LazMan
join:2003-03-26
Angus, ON | reply to TomS_
Loving the in-chassis OOB access!
We're using Raven 1x wireless modems for OOB at some sites, but they are an external "box" - the HWIC option is cool!  |
|
cooldude9919
join:2000-05-29
Cape Girardeau, MO
clubs: | reply to TomS_
lol we still use dial backup for OOB access. Old and slow, but works for the most part |
|
TomS_
debugger it
Premium,MVM
join:2002-07-19
Australia
| I consider dialin OoB to be the more reliable method, and use it whereever I can.
But some of the sites I am rolling out are too far away from any PSTN infrastructure, and/or its simply too expensive to get a PSTN line put in, so 3G it is.
Weve also used some GSM based modems in the past. The cool thing about the 3G solution is that you can maintain IP access to your gear, rather than just dial into a console server. Will be interesting to see how well it works in the real world (seems to work pretty well in the lab). |
|
cow116
Cisco Kid
Premium
join:2003-03-10
Indianapolis, IN
clubs:
 ·RoadRunner Cable
| *grins from a geeky thought* so tom with the HWIC 3g cards in there you're saying that in addition to OOB access you can also use "web access" for example you have a linux or windows box behind them at one of the towers.... fiber connectivity goes down but u still have 3g up so you remote in over the 3g and see whats up over an RDP session?
--
Grambonet Networks
»cowkilla.dyndns.org/cisco |
|
tubbynet
reminds me of the danse russe
Premium
join:2008-01-16
Chandler, AZ
·Cox HSI
 ·Callcentric
 ·Sprint Mobile Broa..
 ·FrontierNet Intern..
| reply to TomS_
TomS_  , are you registering the 3g radio ips using a dynamic dns service? i have thought about the feasibility of using 3g for oob, but even when using my sprint card for extended periods of time i have noticed that my connection will bounce and with it, my ip. or do you have a static ip address with each radios service?
q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..." |
|
TomS_
debugger it
Premium,MVM
join:2002-07-19
Australia
| said by cow116 :fiber connectivity goes down but u still have 3g up so you remote in over the 3g and see whats up over an RDP session? Yes thats basically what Im doing.
said by tubbynet : TomS_ , are you registering the 3g radio ips using a dynamic dns service? I could have static, however, Ive chosen to create a DMVPN between each of the 3G enabled console servers and a central console server, so the IPs are dynamic, and it doesnt really matter what the WAN IP of the 3G service is.
I can share more detail on exactly how I am doing this if you like. |
|
cow116
Cisco Kid
Premium
join:2003-03-10
Indianapolis, IN
clubs: | *giggles* thats awesome |
|
tubbynet
reminds me of the danse russe
Premium
join:2008-01-16
Chandler, AZ
·Cox HSI
·Callcentric
·Sprint Mobile Broa..
·FrontierNet Intern..
1 edit | reply to TomS_
said by TomS_ :I could have static, however, Ive chosen to create a DMVPN between each of the 3G enabled console servers and a central console server, so the IPs are dynamic, and it doesnt really matter what the WAN IP of the 3G service is. I can share more detail on exactly how I am doing this if you like. interesting. i've never really played with dmvpn too much. most of our customers just have a site or two, and in that case they generally just opt for site to site tunnels using asa hardware (especially because the remote sites are easily served by a limited-license asa5505).
i assume that by running a dmvpn, the spoke sites negotiate with the hub to provide the tunnel. once the tunnel is constructed, you are essentially given access to all "internal" addressing behind each site. you then have all routing and data run over your main link, so only management traverses the 3g (hence the oob, part).
very interesting. i'd love to see more about this.
q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..." |
|
cooldude9919
join:2000-05-29
Cape Girardeau, MO
clubs:
1 edit | said by tubbynet :said by TomS_ :I could have static, however, Ive chosen to create a DMVPN between each of the 3G enabled console servers and a central console server, so the IPs are dynamic, and it doesnt really matter what the WAN IP of the 3G service is. I can share more detail on exactly how I am doing this if you like. interesting. i've never really played with dmvpn too much. most of our customers just have a site or two, and in that case they generally just opt for site to site tunnels using asa hardware (especially because the remote sites are easily served by a limited-license asa5505). i assume that by running a dmvpn, the spoke sites negotiate with the hub to provide the tunnel. once the tunnel is constructed, you are essentially given access to all "internal" addressing behind each site. you then have all routing and data run over your main link, so only management traverses the 3g (hence the oob, part). very interesting. i'd love to see more about this. q. We run a ~130 spoke 3 hub dmvpn net work. Spoke to spoke tunnels are made on the fly as needed by the given traffic. Given in our setup all Ip's are static, but it doesnt have to be that way and the spokes can be on a dynamic IP. Works quite well. |
|
joshb
Don't sweat the small stuff.
Premium
join:2006-03-04
Calgary, AB
clubs:
·TELUS
·TekSavvy Solutions..
·Shaw
·Primus Talkbroadband
 ·GoDaddy Hosting
| reply to TomS_
Very Nicely done...Looks good
On a side note,
We are busy staging a system right now at the office and we have so much equipment we have it strung out all over the office... The bench area, a couple office's and the board room... Running big 50+ foot cat5 cables between room...
--
R.I.P Mom We miss you. |
|
TomS_
debugger it
Premium,MVM
join:2002-07-19
Australia
| reply to tubbynet
said by tubbynet :very interesting. i'd love to see more about this. I'll write something up with a diagram to explain how it works. Stay tuned. |
|
tubbynet
reminds me of the danse russe
Premium
join:2008-01-16
Chandler, AZ
·Cox HSI
·Callcentric
·Sprint Mobile Broa..
·FrontierNet Intern..
| said by TomS_ :I'll write something up with a diagram to explain how it works. Stay tuned. you got me on pins and needles!
q.
--
"...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..." |
|
tomdlgns
join:2003-03-21
Chicago, IL | reply to TomS_
that looks somewhat fun. |
|
cow116
Cisco Kid
Premium
join:2003-03-10
Indianapolis, IN
clubs: | reply to TomS_
*bumping* hope the wireless didnt kill yah |
|
TomS_
debugger it
Premium,MVM
join:2002-07-19
Australia
| Oh yeah, I forgot about this. Sorry.
Have drawn the diagram, just needed to put together a post with details of how it works.
Hopefully get that done when I get home tonight. Too busy to do it right now. |
|
TomS_
debugger it
Premium,MVM
join:2002-07-19
Australia
| reply to TomS_
Ok so, referring to the diagram.
The primary goal of the design is to allow IP connectivity to console servers 1..n when the primary network connectivity is down. The primary network connectivity could go down due to the local router failing, an upstream router failing, a network link failing somewhere, etc etc.
The core router distributes a default route, and each of the local routers will pass this on via an OSPF session to their adjacent console server. The subnet which the central console server is also distributed to each console server via OSPF (this becomes important later on).
Each console server also has a loopback interface with a /32 on it. On the core router there are matching static routes with a metric of 255 pointing towards the centra console server so that they dont interfere with routes learned via OSPF.
The central console server then has static routes to each of the other console servers via the appropriate neighbor IP over a multipoint GRE tunnel which is maintained using DMVPN.
So under normal network conditions when the primary backhaul is functional, each console server has a default route and the subnet the central console server is in learned via OSPF, and using OSPF each console server also lets the rest of the network know about its own loopback address. So normally, all traffic will be routing using the primary backhaul.
When the primary network fails somewhere, console servers beyond that point will lose their default route (including the subnet the central console server is in), and the rest of the network will no longer know about those console servers, at which a static via the 3G connection will fire up.
Once the console servers lose their OSPF default route, a static default route with metric 255 then exists via the GRE tunnel which will establish to the central console server. There is also a steering route for the subnet which the central console server lives in which points out of the 3G connection so that the tunnel can establish (important!).
So now that the network has failed somewhere, a console server only knows how to reach the central console server by routing traffic over the 3G connection.
At this point the GRE tunnel establishes using DMVPN.
On the core router you will no longer find an OSPF route for the console servers that are affected. Instead you will find static routes for their loopback IPs pointing to the central console server.
So traffic that then wants to go to the affected console servers will route into the core router, to the central console server, and over the MPGRE tunnel to the appropriate console server.
The console server will then route traffic back via the GRE tunnel to the central console server, through the core router, etc etc.
Thats the basic operation of it. I hope that made sense (Im tired as hell from a 5:30am start this morning).
Let me know if I need to clarify anything. |
|
cow116
Cisco Kid
Premium
join:2003-03-10
Indianapolis, IN
clubs: | thats actually very cool |
|
TomS_
debugger it
Premium,MVM
join:2002-07-19
Australia
| 
Shot of the gear | 
Antenna mounted outside the hut |
Heres a couple more bits of pr0n, because pr0n is good. :D
The tower with the mass of dishes isnt ours, and its not the local mobile cell either. Our tower is the one to the left.
Heres the output of a "sh cell x/x/x radio" command for the HWIC-3G:
Unfortunately, the HWIC-3G doesnt do 3G in 900mhz, but our mobile provider does (and only in rural areas) so were stuck with GPRS, but its still quite reasonable just for console access. |
|
