Communications between internal sites within the same organization is preferred to be delivered in a form of secure or private connection, which ride over some circuit. The circuit could be in the form of dedicated circuit or broadband circuit such as DSL and Cable Internet.
Dedicated Circuit
Dedicated Circuit is a circuit to provide private dedicated connection between two sites or more. In other word, no other organization will use this circuit since it is dedicated to only one organization among its all sites end to end.
Following is the most common dedicated circuit type
1. T1/E1, DS-3
2. ISDN
3. Frame Relay
4. Fiber: OC-X, Metro Ethernet, SONET Ring, DWDM
To have this circuit, usually organization contact its preferred ISP to setup one. The organization could choose to use the ISP network as "intermediate network" between organization sites, or choose to have direct connection between sites bypassing ISP network.
Using T1/E1 circuit for such direct connection for example, the circuit would be some type of leased line; point to point between two sites. When there are more sites to connect, usually organization would use the ISP network at some point to reduce cost and to be more manageable.
This kind of connection technology is considered "top of the line" since it is the most reliable connection (at least for most of the time) compared to broadband connection such as DSL and Cable Internet. This nature requires the organization to pay premium maintenance cost compared to the broadband connection.
Wireless
In some situations, using wireless technology (i.e. microwaves) to provide private site-to-site connection is a good approach. Typically following are the situations that make wireless deployment is a "no-brainer" solution.
• Distance between all sites are pretty closed to each other
• Line of sight (LOS) between antennas are not blocked. In other words; neither trees, hills, mountains, nor buildings are between sites
• You need "unlimited" bandwidth with limited time and budget constraints to deploy
• "Little service abruption" is acceptable
VPN (Virtual Private Network)
With today's virtual communication technology, one organization could use some form of VPN (Virtual Private Network) to provide private and secure site-to-site connection.
Using VPN, connection between two locations could ride over public network (i.e. The Internet) while keep maintaining secure or private connection. This is done by creating logical or virtual connection between the locations that ride over any physical circuit.
There are several technology to set such connection
1. HTTPS/SSL
2. IPSec
3. MPLS
Following is the breakdown.
HTTPS/SSL-based Approach
One factor that contributes to decisions of setting up private or secure connection for internal communications is depending on the application, such as the file transfer and email. Let's say your organization uses web-based email or any web-based application accessible using your Internet browser (such as Internet Explorer, Netscape, or Mozilla) for site inter-communication. When this is the case, then one way of setting up private connection is to utilize HTTPS/SSL-based connection over the Internet.
HTTPS/SSL-based connection is basically HTTP (web) communication that can ride over any connection, including the Internet (public network) via any ISP while still maintain secure and private environment. By utilizing this HTTPS/SSL-based technology approach, any organization sites only need basic Internet connection without require special network setup.
Note that HTTPS/SSL-based network over the Internet only works when all necessary applications within the organizations are web-based applications. Some applications cannot be accessed simply by using Internet browser. For example, you cannot use Internet Explorer (as the Internet browser) to map share drives in Active Directory Microsoft network.
When remote users need to access these applications, then HTTPS/SSL-based approach will not work. To make it work, there would be a need to have network-layer connection technology approach (by go lower to OSI Layer 1 to 3) to setup such secure or private connections.
Using network-layer connection technology approach, any application (web or non-web based) will work since this approach is more general and not depended by specific application types.
IPSec Approach
Both IPSec and HTTPS/SSL technology are VPN connection. They both create encrypted data connection ("tunnel") between two sites. The difference is that HTTPS/SSL is web (OSI Layer 7) approach and IPSec is network (OSI Layer 3) approach.
As mentioned, IPSec VPN is capable of supporting web or non-web applications since it is using network-layer connection technology approach. Example of non-web application is accessing data in Microsoft Active Directory network share drives.
Note:
Both IPSec and HTTPS/SSL VPN technology is also applicable to remote users connecting to office temporary as following description.
Within an organization, there is probably at least one employee that is always "on the run" and need to access work remotely from anywhere. Sometime this type of employee is called "road warrior". There are also other type of employees that need to access work remotely from home, hotels, or any place from time to time.
The nature of such connection need is temporary access, where access is available only when it is needed. When the access is not needed anymore, the access could be closed or removed.
For this nature of remote access, either IPSec or HTTPS/SSL VPN should be a good choice to provide private and secure connection to office/sites; since these VPN technology create "temporary tunnel" between the office and remote users or sites to provide necessary data passing between the locations. When there are no more data passing, the tunnel will be removed.
On implementation, the employees (remote users) could go to the nearest Internet cafe or could use public wireless network to establish IPSec tunnel or HTTPS/SSL to office for work; assuming the employees have necessary tools or equipments.
Between Broadband and Dedicated Circuit
For most small organizations, broadband connection such as DSL and Cable Internet are preferred instead of having dedicated point-to-point circuit due to financial constraint. To provide the private and secure site-to-site connection, such organizations would utilize HTTPS/SSL, IPSec, or both technology.
As illustration, there is a small organization that has two sites. One site has DSL and another has Cable Internet connection. To provide a private an secure site-to-site connection, the organization has a choice to deploy T1 circuit to connect the two sites. Another choice is to deploy IPSec VPN tunnel between sites where each site utilizes the existing broadband connection.
Since the T1 circuit is "more expensive" than the DSL or Cable Internet, the organization then chooses to deploy the second choice. Keep in mind that DSL and cable Internet have lower SLA compared to the dedicated circuit. When the broadband connection is down, the ISP response time will be longer than the dedicated circuit ISP response time.
In addition, these VPN technology could be down "by itself" without obvious reason. Using dedicated circuit, in general the connection is more stable.
MPLS
MPLS is OSI Layer-2/3 VPN approach which is using dedicated point-to-point circuit between organization site to its ISP. Unlike the previous Dedicated Circuit network, MPLS will use the ISP public network that ride over ISP IP-based network devices without deal with the customer IP information. In other word, MPLS approach is somewhat between the Dedicated Circuit approach and IPSec VPN approach.
Generally speaking, ISP network will handle the VPN aspect and use the ISP public network securely and privately; which will be transparent to the organization (the ISP customer) sites. Using MPLS, site-to-site connection is pretty much like the previous dedicated site-to-site connection between sites from the organization perspective.
Network-Layer Site-to-Site Connection Approach
The network-layer site-to-site connection approach refers to IPSec VPN, Dedicated Circuit, and MPLS technology. As mentioned, this network-layer approach is needed to provide connection to the remote sites for any application type including non-web-based applications.
The next discussion will relate to considerations of having such site-to-site connection. Note that these considerations apply to site-to-site connection and do not apply to road-warrior-to-site connection.
Network Topology
When there are only two sites to communicate, the site-to-site connection setup should be just a straight point-to-point. When there are more sites to communicate, there are further considerations to review.
One of the consideration is the network topology. Most common site-to-site network topology setup for three sites or more as follows
1. Full Mesh
2. Hub and Spoke
3. Partially Mesh
Full Mesh
With Full Mesh connection, each site has dedicated connection to each other site as follows:
Site A --- Site B
| \ / |
| \ / |
| \ / |
| X |
| / \ |
| / \ |
| / \ |
Site C --- Site D
Typical organization that employ this connection is organization that has small number of branches or sites with relatively low data throughput.
When the organization has dedicated point-to-point circuits, then there will be (let's say) multiple dedicated T1 connections between sites. Reviewing illustration above, there will be three T1 from one site to others; which make the total of six T1 circuits. When the organization had dedicated VPN tunnels, there will be a total of six tunnels which each site has three tunnels to others.
Since each site has dedicated connection to each other, there will be no single point of failure. If one site is down, other sites still have connections within themselves.
However this kind of setup is considered high cost to manage when number of sites grow and/or larger data throughput are pushed down. With more sites, there will be more dedicated connections to each additional sites.
With dedicated circuits, then there will be more circuits to setup at each site which may be financially prohibitive. With VPN tunnels, then there will be more tunnels to setup which may consume too much VPN device resources such as CPU and memory.
Hub and Spoke
With Hub and Spoke connection, each site will only have a single connection to one central site. This central site then has multiple connections to each other site as follows
Site A
|
|
Site B ---- Site Z ---- Site C
|
|
Site D
Site A to D are called "spoke" and Site Z is called "hub". Note that some people refer this setup as "star topology".
Usually medium to large organizations have this setup. The hub is usually the corporate office and the spokes are branches, smaller offices, or remote offices.
When the organization uses dedicated circuits, there is only a single circuit needed to connect any other sites. With VPN tunnels, the VPN device resources are not consumed much compared to the Full Mesh setup.
The down side is that there is a single point of failure at Site Z (the central site). When this site is down, then all other sites lose connections.
Partially Mesh
Reviewing the two previous setup, you may wonder which the feasible setup that has no single point of failure but not cost prohibitive. The answer is probably the Partially Mesh setup.
With Partially Mesh setup, there will be not much existing connections like Full Mesh; and no single point of failure like Hub and Spoke. Following is illustration.
+------------------+
/ Site A | Site D --------+
/ / \ | / |
/ / \ | / |
Site B --- Site Y ---- Site Z --- Site E |
/ | | \ / \ | |
/ | | \ / \ | |
| | | Site C Site F | |
| | | | | |
| | +--------------+ | |
| | | |
| +------------------------+ |
| |
+---------------------------------+
The Site Y and Site Z are the "hubs". Site A to F are "spokes" to both Site Y and Site Z.
This setup is the preferred one on medium to large organizations. The both hubs are usually two large offices. The spokes are branches, smaller offices, or remote offices.
IP Routing
With either Point-to-Point, Hub and Spoke, Full Mesh, or Partially Mesh network setup; IP routing should be used to interconnect all sites. With this in mind, each site has its own subnet. Router will be used to interconnect sites.
Specifically for IPSec VPN, you could consider to have the router to terminate the VPN tunnel. You could also consider using dedicated VPN box such as firewall or VPN concentrator to provide the VPN tunnel; and use router only to interconnect sites.
Combination of Point-to-Point and Partially Mesh
As mentioned, traditional connection between two sites is just a single point-to-point. However it is possible to have redundant (multiple) point-to-point connection between two sites to provide automatic failover and/or load balance mechanism; where each connection has its own circuit on each site.
Following is the illustration. Let's say there are two sites that have two redundant point-to-point connections between each other. One site has a dedicated point-to-point T1 circuit to the other site and DSL connection. Another site has the other end of dedicated point-to-point T1 circuit and Cable Internet connection. Between the DSL on one site and Cable Internet on the other site, there is a IPSec VPN tunnel connecting the two sites as alternate path of the T1.
With such automatic failover and/or load balance mechanism in mind, following setup could be in place as well.
• Redundant connections between two Hubs in Partially Mesh network
• Redundant connections between one Hub and one Spoke
When there are redundant connections, it means there are multiple path between two sites. Note that with Full Mesh and Partially Mesh network, there are also multiple path between two sites. For such multiple path, dynamic IP routing should be deployed to optimize connections. In addition, packet-based or destination-based load balancing could be considered as well. With hub and spoke setup, static routing should be sufficient.
Starting to Design the Network
When you start designing the network, several aspects come into play
• Circuit choice
• IP address or subnet to use
• Routing protocol to provide connection
Typical network design for site-to-site connection from circuit choice perspective are following
• Dedicated circuit between sites; either uses private point-to-point, frame relay, or MPLS
• Dedicated circuit between sites as primary connection and IPSec VPN tunnel between sites as alternate connection
• IPSec VPN tunnel between sites
For small organizations, it is probably preferable to have full-mesh site-to-site VPN using broadband connection (DSL or Cable Internet) at each location. For simplicity, it is suggested to use the same ISP to provide the broadband connection at all sites. As illustration, all sites could be using Cisco ASA 5505 with 3MBps Cable Internet connection to have the full-mesh site-to-site VPN.
When you choose to have partially mesh or hub and spoke setup (either the circuit or VPN), make sure that the hub has large bandwidth and powerful network device to handle data throughput from other sites. As illustration, the hub could be using Cisco 3825 router with DS-3 circuit where spokes could be using Cisco 1841 router with 1.5MBps DSL connection to have hub-and-spoke site-to-site VPN.
Note:
For more info on Cisco equipment performance, check out the following FAQ
»Cisco Forum FAQ »Cisco Equipment Performance (per pps and Mbps)
Following is illustration. Let's say you decide to use the second choice where there are dedicated circuits between sites as primary connection and IPSec VPN tunnel over the Internet between sites as alternate connection. To start designing the network, you may start to question yourself these and go from there.
• Do you need dedicated equipment for Internet gateway and another for private site-to-site connection?
• Which is the suitable routing protocol to set dedicated circuit as primary connection and to set IPSec VPN tunnel as alternate connection?
• Is there possibility of site-to-site interconnectivity without going over IPSec VPN tunnel eventhough the connection goes over the Internet?
• Which IP address or subnet to use, Private or Public IP address?
• Will there be a NAT/PAT process in place?
• How much budget to spend to cover everything (equipments, circuits, infrastructures, etc.)
• How much connection downtime you can tolerate
• How much data throughput travel across each connection
• How long it takes to test the new network setup
• How immediate you need to have "live" network
Next discussions will view other important aspects.
Network Device Choice
When the organization chooses to use dedicated circuits to have private site-to-site connections, usually the network device would be either router or layer-3 switch where the WAN port would match the circuit specification.
Let's say the circuit would be Frame Relay and the organization selects Cisco router for all sites as the network device. You would use the router WAN port to connect to the Frame Relay circuit. This WAN port should be something like WIC T1 or E1 for internal DSU/CSU or WIC 1T for external DSU/CSU.
If the circuit is Gigabit Ethernet for example, then the network device could be a router or layer-3 switch. In Cisco world, the router could be something like 2821 model; and the layer-3 switch could be something like Catalyst 3750 switch.
When VPN connection is selected to provide the private site-to-site connection, there are also multiple network device alternatives such as router, layer-3 switch, firewall, and VPN concentrator. For small businesses, typical choices are firewall and router. In Cisco world, the firewall is ASA 5500 series and the router is 800 series or higher.
Whichever network device chosen, it is suggested to have the same brand for all of them. When you decide to use Cisco equipments let's say, then all sites should also use Cisco as the network device peer. In theory, multi-vendor equipments are inter-operate-able. However in practice, there are sometime unexpected behaviors when establishing connections between multi-vendor equipments. With single-vendor equipment, network behaviors are more predictable and controllable, leads to more stable network.
Another aspect of having the same-vendor equipments throughout the organization is network administration simplification. Network administrators could concentrate to only a single brand to administer. You don't have to deal with multi vendor when it comes to the network device technical or customer support. You might even receive discounts when you have device large volume number from the same single vendor.
Note:
To guide you in choosing the proper Cisco equipment, check out the following FAQ
»Cisco Forum FAQ »Which Cisco router, switch, VPN, firewall, or else is right for my situation?
Internal and External Connections
All the site interconnections such as file transfer between sites are considered internal connection. External connection is a connection to an outside world, such as connection to server located at the Internet or at external site; or Internet browsing.
For internal connections, the traffic should take the private connection. For external connections, there are multiple choices to consider. One way is to go directly out off the site to the external site. Another way is to go through other internal site before going out to the external site.
Let's review the following situation. Let's say one remote office need to have the updated Microsoft Windows patches. To retrieve the patches, there are several choices. One is to go directly out to the Internet, access the Microsoft sites, and download patches. Another way is to go to central office where the central office run a server that provide updated patches.
For small organizations, usually the preferred way for the remote office to receive the patches is by going directly out to the Internet to retrieve patches. However some situations require the remote office to access the central office's server to retrieve patches.
Should the organization have this second situation, there would probably a need to configure remote office network device to direct traffic to the central office's server for remote office upgrade patch need; and block any attempt from remote office to access the Internet directly to retrieve patches. With this situation, the network is considered more secure since the traffic is more controllable.
Remote Site and Internet Access
As previously mentioned, some situations require remote office to access central office before accessing external sites. However situation such as Internet browsing could not require central office access from remote office perspective. The remote office could just go out to the Internet for Internet browsing.
A good side of accessing the Internet directly without going through central office is that the central office bandwidth is not bogged down by the remote office's Internet traffic. The central office bandwidth then can be conserved for strictly internal access such as file sharing.
The down side of this approach is that the central office probably has no or minimum control of remote office's Internet access activities. Without such control, there is possible security risk or improper use of Internet access such downloading illegal software or virus/worm attack without the central office approval. Therefore for larger organizations, all traffic from remote offices including Internet access must go through central offices for data traffic management, including traffic policing at all sites. Note that from network security and network management perspective, traffic policing at all sites might be considered necessary eventhough it could create network administrative burden.
Keep in mind that it is possible to have the same level of control of remote office Internet access activities as the central offices when those remote offices have their own local Internet connection. With this kind of setup, the organization then has to control multiple Internet connection that are spread among multiple sites (both central and remote offices). Any type of control that take place in central offices must take place in remote offices as well. This is also a common practice for larger organizations. Note that this kind of remote office control might mean additional investment on each remote office to duplicate or to mimic central office.
Whichever the preferred setup, the network administrator should consider the trade offs between the two setup choices. For small business, direct Internet access from remote offices could be the preferred choice. When the organization is concerned more on the network security, then the organization might consider the second setup choice.
IPSec VPN and Internet (External Connection) Access
Let's say an organization permit their remote offices to go out to the Internet directly without going through central office. Typically there would be two separate connections at the remote office. One is to serve the internal access and another is to serve the Internet access.
Specifically for organizations that use IPSec VPN connections to serve the site inter-communication, there should be some kind of split tunneling to provide the separate connections between the Internet access and internal access. For Internet access, typically PAT (Port Address Translation) is used to bridge Private Subnet used in internal network (LAN) and the Internet. Using PAT; application traffic that use the most common IP protocol such as TCP, UDP (and ICMP) from local LAN are PAT-ed to the Public IP address.
Let's review the IPSec VPN tunnel setup requirement. IPSec tunnel would use IP Protocol 50 (ESP) or 51 (AH) to setup the VPN tunnel. Unlike TCP and UDP, ESP and AH have no concept of port numbers; hence in theory, these security protocols cannot be PAT-ed.
Should the organization permit remote offices to go out to the Internet directly and the organization deploys VPN tunnel to serve internal access; then each site should have at least two Public IP addresses. One IP address would serve the Internet access (to be PAT-ed as many as needed) and another IP address would be reserved for the VPN peer to other sites (or for any IP protocols that are un-PAT-able).
For small business, it is probably preferable to have each site having those two Public IP addresses assigned to the same gateway (or peer) network device, which then the traffic will ride over the same circuit. For medium or large business that quite large number of sites, each Public IP address could reside at different network device and could ride over different circuit.
Name Resolution
In sharing files between sites, the organization might use DNS server to resolve name to IP addresses. When the organization deploys Microsoft network, then there might also be WINS server in addition to the DNS server.
Let's say the organization permit remote office to go out to the Internet directly without going through the central office. The preferred way is to have the remote office to use the local ISP DNS server to reach the Internet sites. For internal access, the remote office uses internal DNS server to reach internal servers. The unwanted setup is to have the remote office to use the central office's internal DNS server to access the Internet since it will bog down the central office's bandwidth.
To have the preferred way, there are alternatives to setup the DNS/WINS servers at remote offices. One way is to setup local DNS/WINS server at each remote site. With this setup, any traffic (internal or external traffic) from remote office will use the local DNS/WINS server. The central office's DNS/WINS servers will be used only if the traffic are internal. When the traffic are external, only ISP DNS server will be used. The external traffic from remote office will never go through the central office. The down side is that this setup is probably cost prohibitive, not to mention network administration prohibitive.
Another way to setup is to assign multiple DNS/WINS IP addresses at remote site hosts. Assign both central office's DNS/WINS servers and also assign the remote site's local ISP DNS IP addresses to all remote site hosts. In addition, there might be a need to create traffic filtering on the remote office's network device to allow name resolving traffic to use central office's DNS/WINS server only when the traffic are internal; and to block attempted central office's DNS/WINS server access for external traffic. Similarly, there would be traffic filtering to allow name resolving traffic to use the local ISP DNS IP address only when the traffic are external. With this setup, there should be no need to deploy DNS/WINS servers at each remote site to provide name resolving and still be able to avoid central office bandwidth bogged down by the remote office's external traffic.
Real Network Illustration
Check out the following threads for illustration
»IPsec help 1811
»[HELP] BGP Failover to IPSEC
»How to Loadshare between a E1 LInka nd Ebgp(MPLS) Link
Deployment Process
Check out the following FAQ for following topics in network design
1. Between Hub and Spoke, Full Mesh, and Partially Mesh
»Cisco Forum FAQ »Tips in Designing Network on Hub-and-Spoke, Full-Mesh, or Partially-Mesh setup
2. VPN
»Cisco Forum FAQ »Between GRE/IPSEC and IPSEC VPN tunnels
»Cisco Forum FAQ »Various Site-to-Site IPSec VPN: Cisco, Juniper, Checkpoint, Sonicwall, Zywall
»Cisco Forum FAQ »Private Routing over VPN: GRE over IP Sec
 show feedback form
 close
by aryoba 
last modified: 2008-04-16 15:25:19 |
