L3VPN – Hub & Spoke VPN: Two Interfaces

1. Intro

In an MPLS L3VPN, Hub-and-Spoke is a VPN topology where spoke sites cannot communicate directly with each other. Traffic between spokes must pass through the hub CE. This is commonly used when the hub provides centralized firewalling, Internet access, shared services, security inspection, or centralized routing.

The are two main categories for configuring a Hub & Spoke VPN:

One-interface H&S
Two-interfaces H&S

In this post we will focus on the Two-interface H&S model. For the One-interface H&S model visit the previous post.

Two-Interface H&S VPN:

two interfaces between Hub-CE and Hub-PE.
each interface will be in its own VRF.
- one Hub VRF is receiving routes from spokes.
- the other Hub VRF is advertising routes to spokes.
between spokes, specific routes are exchanged only through the hub, therefore traffic will also pass through HUB.
asymmetric routing and traffic forwarding.

In our scenario we have a customer, we will call it ‘HS‘ customer, and it wants a Hub & Spoke VPN (with one interface at the hub), the topology is the following:

In this typical two interfaces H&S VPN we can see the asymmetry that was mentioned above:

We will use 2 x Route Targets (RTs) and but also 2 x VRFs at the HUB PE.
The Spoke VRF will export to one HUB VRF and will import from the other HUB VRF.
Similarly, one HUB VRF will only import routes from Spokes, while the other HUB VRF will only export routes to Spokes.
At the HUB site each VRF has its own interface to HUB-CE, and because of the routing asymmetry, we will also have a traffic asymmetry, where one PE-CE interface will only receive traffic, while the other will only send traffic.

In more details:

All Spokes will export routes to Hub-import-from-Spokes
VRF Hub-import-from-Spokes will import Spoke routes.
VRF Hub-import-from-Spokes will export nothing.
HUB-CE will receive the routes from VRF Hub-import-from-Spokes via one interface.
HUB-CE will advertise the routes to VRF HUB-export-to-Spokes via the other interface.
VRF HUB-export-to-Spokes will import nothing.
VRF HUB-export-to-Spokes will export routes to spokes.
All Spokes will import routes from the VRF HUB-export-to-Spokes.

If we translate this to Route Targets, it will look like this:

VRF Spoke
- vrf-export target:64512:302
- vrf-import target:64512:301
VRF Hub-import-from-Spokes
- vrf-import target:64512:302
- vrf-export NULL
- interface ge-0/0/4.311 (PE-CE)
VRF HUB-export-to-Spokes
- vrf-import NULL
- vrf-export target:64512:301
- interface ge-0/0/4.310 (PE-CE)

2. Configuring H&S VPN

Considering the details from the introduction, let’s see the configuration:

### Spoke config (R7 and R8 have similar config): 

set routing-instances HS-SPOKE instance-type vrf
set routing-instances HS-SPOKE protocols bgp group PE-CE peer-as 64512
set routing-instances HS-SPOKE protocols bgp group PE-CE neighbor 172.16.0.5
set routing-instances HS-SPOKE interface ge-0/0/4.317
set routing-instances HS-SPOKE interface lo0.101
set routing-instances HS-SPOKE route-distinguisher 17.0.0.7:312
set routing-instances HS-SPOKE vrf-import HS-Hub
set routing-instances HS-SPOKE vrf-export HS-Spoke
set routing-instances HS-SPOKE vrf-table-label

set policy-options policy-statement HS-Hub term IMPORT from community CM-HS-Hub
set policy-options policy-statement HS-Hub term IMPORT then accept
set policy-options policy-statement HS-Hub then reject
set policy-options policy-statement HS-Spoke then community add CM-HS-Spoke
set policy-options policy-statement HS-Spoke then accept

set policy-options community CM-HS-Hub members target:64512:301
set policy-options community CM-HS-Spoke members target:64512:302

### HUB config: 

set routing-instances HS-HUB-export-to-Spokes instance-type vrf
set routing-instances HS-HUB-export-to-Spokes protocols bgp group HUB-CE peer-as 64512
set routing-instances HS-HUB-export-to-Spokes protocols bgp group HUB-CE neighbor 172.16.0.1
set routing-instances HS-HUB-export-to-Spokes interface ge-0/0/4.311
set routing-instances HS-HUB-export-to-Spokes route-distinguisher 17.0.0.1:310
set routing-instances HS-HUB-export-to-Spokes vrf-import NULL
set routing-instances HS-HUB-export-to-Spokes vrf-export HS-Hub-export
set routing-instances HS-HUB-export-to-Spokes vrf-table-label

set routing-instances HS-HUB-import-from-Spokes instance-type vrf
set routing-instances HS-HUB-import-from-Spokes protocols bgp group HUB-CE peer-as 64512
set routing-instances HS-HUB-import-from-Spokes protocols bgp group HUB-CE neighbor 172.16.0.3
set routing-instances HS-HUB-import-from-Spokes interface ge-0/0/4.312
set routing-instances HS-HUB-import-from-Spokes route-distinguisher 17.0.0.1:311
set routing-instances HS-HUB-import-from-Spokes vrf-import HS-Spoke-import
set routing-instances HS-HUB-import-from-Spokes vrf-export NULL
set routing-instances HS-HUB-import-from-Spokes vrf-table-label

set policy-options policy-statement HS-Hub-export then community add CM-HS-Hub
set policy-options policy-statement HS-Hub-export then accept
set policy-options policy-statement HS-Spoke-import term IMPORT from community CM-HS-Spoke
set policy-options policy-statement HS-Spoke-import term IMPORT then accept
set policy-options policy-statement HS-Spoke-import then reject
set policy-options policy-statement NULL then reject

set policy-options community CM-HS-Hub members target:64512:301
set policy-options community CM-HS-Spoke members target:64512:302

Note: The configuration is not complete yet, we still need to solve the AS loop prevention issue, as we will see bellow.

The important part is to understand the VRF Route Target configuration and how it works in this scenario.

Now let’s see how the R8 router is adverting a route to R7 (via the HUB):

### On R8 we are receiving prefixes from the CE: 

root@R8# run show route receive-protocol bgp 172.16.0.7 table HS-SPOKE.inet.0  

HS-SPOKE.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
* 192.168.8.0/24          172.16.0.7                              64512 I

### Via the Route Reflector we are advertising the route to R1: 

root@R8# run show route advertising-protocol bgp 17.0.0.4 table HS-SPOKE.inet.0 192.168.8.0/24 detail    

HS-SPOKE.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
* 192.168.8.0/24 (1 entry, 1 announced)
 BGP group iBGP type Internal
     Route Distinguisher: 17.0.0.8:312
     VPN Label: 40
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [100] 64512 I 
     Communities: target:64512:302

### We see the route on R1: 

root@R1# run show route table HS-HUB-import-from-Spokes 192.168.8.0/24 detail 

HS-HUB-import-from-Spokes.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
192.168.8.0/24 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 17.0.0.8:312
                Next hop type: Indirect, Next hop index: 0
                Address: 0x7bbff44
                Next-hop reference count: 9, key opaque handle: 0x0, non-key opaque handle: 0x0
                Kernel Table Id: 0
                Source: 17.0.0.4
                Next hop type: Router, Next hop index: 731
                Next hop: 17.1.3.3 via ge-0/0/2.0, selected
                Label operation: Push 40, Push 801008(top)
                Label TTL action: prop-ttl, prop-ttl(top)
                Load balance label: Label 40: None; Label 801008: None; 
                Label element ptr: 0xa3f0370
                Label parent element ptr: 0xa3efa88
                Label element references: 1
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 153
                Protocol next hop: 17.0.0.8
                Label operation: Push 40
                Label TTL action: prop-ttl
                Load balance label: Label 40: None; 
                Indirect next hop: 0x791e218 1048576 INH Session ID: 359
                State: <Secondary Active Int Ext ProtectionCand>
                Local AS:   100 Peer AS:   100
                Age: 32:02      Metric2: 30 
                Validation State: unverified 
                ORR Generation-ID: 0 
                Task: BGP_100.17.0.0.4
                Announcement bits (2): 0-BGP_RT_Background 1-KRT 
                AS path: 64512 I  (Originator)
                Cluster list:  17.0.0.4
                Originator ID: 17.0.0.8
                Communities: target:64512:302
                Import Accepted
                VPN Label: 40
                Localpref: 100
                Router ID: 17.0.0.4
                Primary Routing Table: bgp.l3vpn.0
                Thread: junos-main 

### But we are not sending the route to the HUB-CE: 

root@R1# run show route advertising-protocol bgp 172.16.0.3 table HS-HUB-import-from-Spokes.inet.0 192.168.8.0/24 detail 

[edit]
root@R1#

Because the CE routers are using AS 64512 on all sites, the AS loop prevention mechanism will block advertisements. We can solve this issue in two ways:

advertise-peer-as and loops combo
as-override

### We will configure advertise-peer-as and loops:

set routing-instances HS-HUB-import-from-Spokes protocols bgp group HUB-CE advertise-peer-as 
set routing-options autonomous-system loops 2

### We can see that our route is now advertised to the HUB-CE: 

root@R1# run show route advertising-protocol bgp 172.16.0.3 table HS-HUB-import-from-Spokes.inet.0 192.168.8.0/24 detail 

HS-HUB-import-from-Spokes.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
* 192.168.8.0/24 (1 entry, 1 announced)
 BGP group HUB-CE type External
     Nexthop: Self
     AS path: [100] 64512 I 
     Communities: target:64512:302

Note: Every router will do an AS loop prevention check, and possibly block the routing. What we have here is not the typical AS path loop issue, because in this scenario both the Customer (AS64512) and Provider (AS100) will create AS path loops. This means we will need to apply advertise-peer-as and loops on multiple routers: HUB-CE, Spoke-CE, PEs, even on the provider Route Reflector. Alternatively, as-override can be used, but also this one must be carefully configured on multiple routers.

Note: routing-options autonomous-system loops 2 has a global effect, so alternatively we can also go with:

set routing-instances HS-HUB-import-from-Spokes protocols bgp group HUB-CE family inet unicast loops 3

Back to our checks:

### We are receiving the route from the HUB-CE on the other VRF:

root@R1# run show route receive-protocol bgp 172.16.0.1 table HS-HUB-export-to-Spokes 192.168.8.0/24 detail 

HS-HUB-export-to-Spokes.inet.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
* 192.168.8.0/24 (1 entry, 1 announced)
     Accepted
     Nexthop: 172.16.0.1
     AS path: 64512 100 64512 I 
     Communities: target:64512:302

### This means that the HUB-CE was configured with loops and advertise-peer-as: 
### (ignore the logical-system, just a simpler way to setup the lab)

root@R9# show logical-systems HS-HUB | display set 
set logical-systems HS-HUB interfaces ge-0/0/1 unit 311 vlan-id 311
set logical-systems HS-HUB interfaces ge-0/0/1 unit 311 family inet address 172.16.0.1/31
set logical-systems HS-HUB interfaces ge-0/0/1 unit 312 vlan-id 312
set logical-systems HS-HUB interfaces ge-0/0/1 unit 312 family inet address 172.16.0.3/31
set logical-systems HS-HUB interfaces lo0 unit 300 family inet address 192.168.100.1/24
set logical-systems HS-HUB protocols bgp group HUB-CE advertise-peer-as
set logical-systems HS-HUB protocols bgp group HUB-CE export BGP-EXPORT
set logical-systems HS-HUB protocols bgp group HUB-CE peer-as 100
set logical-systems HS-HUB protocols bgp group HUB-CE neighbor 172.16.0.0
set logical-systems HS-HUB protocols bgp group HUB-CE neighbor 172.16.0.2
set logical-systems HS-HUB policy-options policy-statement BGP-EXPORT from route-filter 192.168.0.0/16 prefix-length-range /24-/24
set logical-systems HS-HUB policy-options policy-statement BGP-EXPORT then accept
set logical-systems HS-HUB routing-options autonomous-system 64512
set logical-systems HS-HUB routing-options autonomous-system loops 3

### Via the Route Reflector we advertising prefixes back to Spokes: 

root@R1# run show route advertising-protocol bgp 17.0.0.4 192.168.8.0 detail 

HS-HUB-export-to-Spokes.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
* 192.168.8.0/24 (1 entry, 1 announced)
 BGP group iBGP type Internal
     Route Distinguisher: 17.0.0.1:310
     VPN Label: 39
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [100] 64512 100 64512 I 
     Communities: target:64512:301 target:64512:302

root@R1# run show route advertising-protocol bgp 17.0.0.4 192.168.100.0 detail 

HS-HUB-export-to-Spokes.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
* 192.168.100.0/24 (1 entry, 1 announced)
 BGP group iBGP type Internal
     Route Distinguisher: 17.0.0.1:310
     VPN Label: 39
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [100] 64512 I 
     Communities: target:64512:301

### On R7 we receive the prefixes from the RR and advertise them to the CE: 

root@R7# run show route receive-protocol bgp 17.0.0.4 table HS-SPOKE.inet.0 detail 192.168.8.0/24 

HS-SPOKE.inet.0: 7 destinations, 8 routes (7 active, 0 holddown, 0 hidden)
* 192.168.8.0/24 (1 entry, 1 announced)
     Import Accepted
     Route Distinguisher: 17.0.0.1:310
     VPN Label: 39
     Nexthop: 17.0.0.1
     Localpref: 100
     AS path: 64512 100 64512 I  (Originator)
     Cluster list:  17.0.0.4
     Originator ID: 17.0.0.1
     Communities: target:64512:301 target:64512:302

root@R7# run show route receive-protocol bgp 17.0.0.4 table HS-SPOKE.inet.0 detail 192.168.100.0/24  

HS-SPOKE.inet.0: 7 destinations, 8 routes (7 active, 0 holddown, 0 hidden)
* 192.168.100.0/24 (1 entry, 1 announced)
     Import Accepted
     Route Distinguisher: 17.0.0.1:310
     VPN Label: 39
     Nexthop: 17.0.0.1
     Localpref: 100
     AS path: 64512 I  (Originator)
     Cluster list:  17.0.0.4
     Originator ID: 17.0.0.1
     Communities: target:64512:301

root@R7# run show route advertising-protocol bgp 172.16.0.5 table HS-SPOKE.inet.0 detail 192.168.8.0/24

HS-SPOKE.inet.0: 7 destinations, 8 routes (7 active, 0 holddown, 0 hidden)
* 192.168.8.0/24 (1 entry, 1 announced)
 BGP group PE-CE type External
     Nexthop: Self
     AS path: [100] 64512 100 64512 I 
     Communities: target:64512:301 target:64512:302

root@R7# run show route advertising-protocol bgp 172.16.0.5 table HS-SPOKE.inet.0 detail 192.168.100.0/24

HS-SPOKE.inet.0: 7 destinations, 8 routes (7 active, 0 holddown, 0 hidden)
* 192.168.100.0/24 (1 entry, 1 announced)
 BGP group PE-CE type External
     Nexthop: Self
     AS path: [100] 64512 I 
     Communities: target:64512:301

### And finally, we have the full end to end path on the Spoke CE. 
### Also, notice how many loops we have in the AS path:

root@R9# run show route 192.168/16 logical-system HS-CE-R7       

inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.7.0/24     *[Direct/0] 02:06:36
                    >  via lo0.317
192.168.7.1/32     *[Local/0] 02:06:36
                       Local via lo0.317
192.168.8.0/24     *[BGP/170] 00:28:54, localpref 100
                      AS path: 100 64512 100 64512 I, validation-state: unverified
                    >  to 172.16.0.4 via ge-0/0/5.317
192.168.100.0/24   *[BGP/170] 00:10:32, localpref 100
                      AS path: 100 64512 I, validation-state: unverified
                    >  to 172.16.0.4 via ge-0/0/5.317

### And while routes follow one path, traffic flows in the opposite direction: 

root@R9# run ping logical-system HS-CE-R7 192.168.8.1 source 192.168.7.1 
PING 192.168.8.1 (192.168.8.1): 56 data bytes
64 bytes from 192.168.8.1: icmp_seq=0 ttl=55 time=11.253 ms
64 bytes from 192.168.8.1: icmp_seq=1 ttl=55 time=10.916 ms
^C
--- 192.168.8.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 10.916/11.085/11.253/0.168 ms

### If we start a fast ping: 

root@R9# run ping logical-system HS-CE-R7 192.168.8.1 source 192.168.7.1 rapid count 10000 
PING 192.168.8.1 (192.168.8.1): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

### On R1 we can see the traffic counters, indicating that the traffic is flowing via HUB: 

root@R1# run monitor interface ge-0/0/4.312

Interface: ge-0/0/4.312, Enabled, Link is Up
Flags: SNMP-Traps 0x4000
Encapsulation: ENET2
VLAN-Tag [ 0x8100.312 ]
Local statistics:                                                Current delta
  Input bytes:                     62649                                   [0]
  Output bytes:                    81583                                   [0]
  Input packets:                    1020                                   [0]
  Output packets:                   1023                                   [0]
Remote statistics:
  Input bytes:                    307020 (119336 bps)                  [93996]
  Output bytes:                      168 (0 bps)                           [0]
  Input packets:                    3655 (177 pps)                      [1119]
  Output packets:                      2 (0 pps)                           [0]
Traffic statistics:
  Input bytes:                    369669                               [93996]
  Output bytes:                    81751                                   [0]
  Input packets:                    4675                                [1119]
  Output packets:                   1025                                   [0]
Protocol: inet, MTU: 1500, Flags: Is-Primary

Note

In our scenario we used BGP as a PE-CE protocol, and the solution to overcome the AS path loop prevention mechanism was the advertise-peer-as and loops combo, but we can also use as-override.

But, what if we have OSPF as a PE-CE protocol? Well, in that case we also have complications, the DN bit that is automatically activated by the PE. The solution will the following:

set routing-instances HS-HUB-export-to-Spokes protocols ospf domain-id disable
set routing-instances HS-HUB-export-to-Spokesprotocols ospf domain-vpn-tag 0
set routing-instances HS-HUB-import-from-Spokes protocols ospf domain-id disable
set routing-instances HS-HUB-import-from-Spokes protocols ospf domain-vpn-tag 0

In this H&S scenario, up/down bit was also an issue for ISIS at the HUB site, and I was not able to make it work.

External Resources:

1. Intro

2. Configuring H&S VPN

Note

Related Posts