Dual ISP and VPN as MPLS back

Working on a task:

1. Use 2 ISPs as Internet connections, one as active and the other as backup.
2. Use VPN using backup ISP as the alternative way when MPLS circuit down.

Did some research and found useful articles:

ASA/PIX 7.x: Redundant or Backup ISP Links Configuration Example
Configuring LAN-to-LAN VPNs
Ken Felix Security Blog -- Cisco ASA ikev2 setup

Setup a lab to simulate these 2 scenarios:

-.logical connection.-


-.physical connection.-

1. Dual ISP portion
The challenge I faced is not when the primary Internet circuit down, default route using backup ISP router SW2. Track object work perfectly.
The real challenge is when the primary circuit restored, because the default route is still pointing to backup ISP router SW2, the tracking object will not be reachable via the ASA interface (in this example is the outside1 interface). This is the way how ASA works.

The trick is to configure a host route of ping target via the primary interface. No matter whether the primary circuit is down, the ASA1 will always use the host route to ping the ping target.

Here are the configuration:

1.1 ASA1
In this example, I actually use 12.12.12.22 as my ping target (that is because I use 12.12.12.4 in the next scenario)

sla monitor 100
type echo protocol ipIcmpEcho 12.12.12.22 interface outside1
num-packets 3
frequency 10
sla monitor schedule 100 life forever start-time now
!
track 100 rtr 100 reachability

This the default route using primary ISP router SW1

route outside1 0.0.0.0 0.0.0.0 1.1.1.11 1 track 100

!This the 2nd default route using secondary ISP router SW2

route outside2 0.0.0.0 0.0.0.0 2.2.2.22 50

In this example, I actually use 12.12.12.22 as my ping target, so configure the following host route to force ASA1 use outside1 to reach the ping target 12.12.12.22

route outside1 12.12.12.22 255.255.255.255 1.1.1.11 1

NAT configuration:

object network office_subnets1
nat (inside,outside1) dynamic interface
object network office_subnets2
nat (inside,outside2) dynamic interface

1.2 SW1

ip route 0.0.0.0 0.0.0.0 12.12.12.4

1.3 SW2
In this lab environment, I have to also force secondary ISP router SW2 to use SW1 to reach 1.1.1.0/24 network.

ip route 0.0.0.0 0.0.0.0 12.12.12.4
ip route 1.1.1.0 255.255.255.0 12.12.12.11

1.4 Validation

1.4.1 Both primary and secondary Internet circuit are UP:
ASA1's partial routing table:

lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1

ASA1 tracking object 12.12.12.22 status:

lab-ASA1/pri/act# sh track 100
Track 100
Response Time Reporter 100 reachability
Reachability is Up
22 changes, last change 00:00:03
Latest operation return code: OK
Latest RTT (millisecs) 4
Tracked by:
STATIC-IP-ROUTING 0

ASA3 can ping 12.12.12.4:

lab-ASA3# PING 12.12.12.4 rep 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 12.12.12.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

1.4.2 When primary Internet circuit is down:

lab-sw(config)#int f1/0/33
lab-sw(config-if)#shut

ASA1 partial routing table:

lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 2.2.2.22 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [50/0] via 2.2.2.22, outside2
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1

ASA1 tracking object 12.12.12.22 status:

lab-ASA1/pri/act# sh track 100
Track 100
Response Time Reporter 100 reachability
Reachability is Down
21 changes, last change 00:00:27
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0

ASA3 can ping 12.12.12.4:

lab-ASA3# PING 12.12.12.4 rep 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 12.12.12.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

1.4.3 When primary Internet circuit restore:

lab-sw(config)#int f1/0/33
lab-sw(config-if)#no shut

ASA1 tracking object 12.12.12.22 status:

lab-ASA1/pri/act# sh track 100
Track 100
Response Time Reporter 100 reachability
Reachability is Up
22 changes, last change 00:00:03
Latest operation return code: OK
Latest RTT (millisecs) 4
Tracked by:
STATIC-IP-ROUTING 0

ASA1's partial routing table:

lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1

ASA3 can ping 12.12.12.4:

lab-ASA3# PING 12.12.12.4 rep 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 12.12.12.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Side note #1:
All the ASA in this lab are running version 9.2(2)4.
The tracking object feature is very limited.
Each track can only include one sla.
It will be ideal if each track can include multiple sla. I will like to have multiple ping targets for my Internet circuit. Rely on one ping target can be very risky.

Side note #2:
If configure:
route outside1 0.0.0.0 0.0.0.0 1.1.1.1 1 track 1
route outside1 0.0.0.0 0.0.0.0 1.1.1.1 2 track 2
This is a "OR" operation.
When track 1 is down, the 2nd route will be installed;
When track 2 is down, the 1st route will be installed;
When both track 1 and 2 are down, the default route will be withdrew.

2. VPN as MPLS backup portion

2.1 ASA1
ASA1 tracking object 210:

sla monitor 210
type echo protocol ipIcmpEcho 192.168.44.44 interface inside
num-packets 3
frequency 10
sla monitor schedule 210 life forever start-time now
!
track 210 rtr 210 reachability

Static routes:

route inside 192.168.44.0 255.255.255.0 192.168.123.3 1 track 210

! host route for ping target

route inside 192.168.34.4 255.255.255.255 192.168.123.3 1

VPN configuration:

crypto ipsec ikev2 ipsec-proposal ESP-AES-SHA512
protocol esp encryption aes
protocol esp integrity sha-512
!
crypto map L2LVPN 10 match address acl_office_to_prod_vpn
crypto map L2LVPN 10 set peer 12.12.12.4
crypto map L2LVPN 10 set ikev2 ipsec-proposal ESP-AES-SHA512
crypto map L2LVPN interface outside2
!
crypto isakmp identity address
!
crypto ikev2 policy 10
encryption aes
integrity sha512
group 2
prf sha
lifetime seconds 28800
crypto ikev2 enable outside2
!
tunnel-group 12.12.12.4 type ipsec-l2l
tunnel-group 12.12.12.4 ipsec-attributes
ikev2 remote-authentication pre-shared-key password
ikev2 local-authentication pre-shared-key password
!
object-group network Office
network-object 192.168.123.0 255.255.255.0
!
object-group network production_subnts
network-object 192.168.44.0 255.255.255.0
!
access-list acl_office_to_prod_vpn extended permit ip object-group Office object-group production_subnts
!
nat (inside,outside2) source static Office Office destination static production_subnts production_subnts

2.2 ASA4
ASA4 tracking object 210:

sla monitor 210
type echo protocol ipIcmpEcho 192.168.123.1 interface c3
num-packets 3
frequency 10
sla monitor schedule 210 life forever start-time now
!
track 210 rtr 210 reachability

Static routes:

route c3 192.168.0.0 255.255.0.0 192.168.34.3 1 track 210
route c3 192.168.0.0 255.255.0.0 12.12.12.22 50

route internet 0.0.0.0 0.0.0.0 12.12.12.11 1

! host route for the VPN end system, make it use ASA1's outside2 interface, not ASA1's outside1

route internet 2.2.2.1 255.255.255.255 12.12.12.22 1

! host route for ping target

route c3 192.168.123.1 255.255.255.255 192.168.34.3 1

VPN configuration:

crypto ipsec ikev2 ipsec-proposal ESP-AES-SHA512
protocol esp encryption aes
protocol esp integrity sha-512
!
crypto map L2LVPN 10 match address acl_prod_to_office_vpn
crypto map L2LVPN 10 set peer 2.2.2.1
crypto map L2LVPN 10 set ikev2 ipsec-proposal ESP-AES-SHA512
crypto map L2LVPN interface internet
!
crypto isakmp identity address
!
crypto ikev2 policy 10
encryption aes
integrity sha512
group 2
prf sha
lifetime seconds 28800
crypto ikev2 enable internet
!
tunnel-group 2.2.2.1 type ipsec-l2l
tunnel-group 2.2.2.1 ipsec-attributes
ikev2 remote-authentication pre-shared-key password
ikev2 local-authentication pre-shared-key password
!
object-group network Office
network-object 192.168.123.0 255.255.255.0
!
object-group network production_subnts
network-object 192.168.44.0 255.255.255.0
!
access-list acl_prod_to_office_vpn extended permit ip object-group Office object-group production_subnts
!

2.3 Validation

2.3.1 When MPLS is UP:
ASA1 partial routing table:

lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
S 192.168.34.4 255.255.255.255 [1/0] via 192.168.123.3, inside
S 192.168.44.0 255.255.255.0 [1/0] via 192.168.123.3, inside
S 192.168.44.44 255.255.255.255 [1/0] via 192.168.123.3, inside
C 192.168.123.0 255.255.255.0 is directly connected, inside
L 192.168.123.1 255.255.255.255 is directly connected, inside

ASA1 tracking object 210 status:

lab-ASA1/pri/act# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
16 changes, last change 00:02:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0

ASA1 IPSec ISAKMP status:

lab-ASA1/pri/act# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs

ASA1 IPSec IKEV2 status:

lab-ASA1/pri/act# sh crypto ikev2 sa
There are no IKEv2 SAs

ASA4 routing table

lab-ASA4# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 12.12.12.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 12.12.12.11, internet
S 2.2.2.1 255.255.255.255 [1/0] via 12.12.12.22, internet
C 12.12.12.0 255.255.255.0 is directly connected, internet
L 12.12.12.4 255.255.255.255 is directly connected, internet
S 192.168.0.0 255.255.0.0 [1/0] via 192.168.34.3, c3
C 192.168.34.0 255.255.255.0 is directly connected, c3
L 192.168.34.4 255.255.255.255 is directly connected, c3
C 192.168.44.0 255.255.255.0 is directly connected, production
L 192.168.44.4 255.255.255.255 is directly connected, production
S 192.168.123.1 255.255.255.255 [1/0] via 192.168.34.3, c3

ASA4 tracking object 210 status:

lab-ASA4# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
14 changes, last change 00:03:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0

ASA4 IPSec ISAKMP status:

lab-ASA4# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs

ASA4 IPSec IKEV2 status:

lab-ASA4# sh crypto ikev2 sa
There are no IKEv2 SAs

ASA3 can ping 192.168.44.44

lab-ASA3# ping 192.168.44.44
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.44.44, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

2.3.2 When MPLS is DOWN:

lab-sw(config)#INT F1/0/44
lab-sw(config-if)#SHUT

ASA1 partial routing table:

lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1
C 172.16.0.0 255.255.255.252 is directly connected, failover
L 172.16.0.1 255.255.255.255 is directly connected, failover
S 192.168.34.4 255.255.255.255 [1/0] via 192.168.123.3, inside
S 192.168.44.0 255.255.255.0 [50/0] via 2.2.2.22, outside2
S 192.168.44.44 255.255.255.255 [1/0] via 192.168.123.3, inside
C 192.168.123.0 255.255.255.0 is directly connected, inside
L 192.168.123.1 255.255.255.255 is directly connected, inside

ASA1 tracking object 210

lab-ASA1/pri/act# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Down
17 changes, last change 00:00:00
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0

ASA4 routing table:

lab-ASA4# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 12.12.12.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 12.12.12.11, internet
S 2.2.2.1 255.255.255.255 [1/0] via 12.12.12.22, internet
C 12.12.12.0 255.255.255.0 is directly connected, internet
L 12.12.12.4 255.255.255.255 is directly connected, internet
S 192.168.0.0 255.255.0.0 [50/0] via 12.12.12.22, c3
C 192.168.34.0 255.255.255.0 is directly connected, c3
L 192.168.34.4 255.255.255.255 is directly connected, c3
C 192.168.44.0 255.255.255.0 is directly connected, production
L 192.168.44.4 255.255.255.255 is directly connected, production
S 192.168.123.1 255.255.255.255 [1/0] via 192.168.34.3, c3

ASA3 tracking object 210

lab-ASA4# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Down
15 changes, last change 00:00:38
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0

ASA3 can ping 192.168.44.44:

lab-ASA3# ping 192.168.44.44
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.44.44, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

ASA1 IPsec ISAKMP status:

lab-ASA1/pri/act# sh crypto isakmp sa
There are no IKEv1 SAs
IKEv2 SAs:
Session-id:6, Status:UP-ACTIVE, IKE count:1, CHILD count:1
Tunnel-id Local Remote Status Role
219900601 2.2.2.1/500 12.12.12.4/500 READY INITIATOR
Encr: AES-CBC, keysize: 128, Hash: SHA512, DH Grp:2, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/147 sec
Child sa: local selector 192.168.123.0/0 - 192.168.123.255/65535
remote selector 192.168.44.0/0 - 192.168.44.255/65535
ESP spi in/out: 0x656293c/0x2a61c4bb 

ASA1 IPsec IKEV2 status:

lab-ASA1/pri/act# sh crypto ikev2 sa
IKEv2 SAs:
Session-id:6, Status:UP-ACTIVE, IKE count:1, CHILD count:1
Tunnel-id Local Remote Status Role
219900601 2.2.2.1/500 12.12.12.4/500 READY INITIATOR
Encr: AES-CBC, keysize: 128, Hash: SHA512, DH Grp:2, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/140 sec
Child sa: local selector 192.168.123.0/0 - 192.168.123.255/65535
remote selector 192.168.44.0/0 - 192.168.44.255/65535
ESP spi in/out: 0x656293c/0x2a61c4bb 

2.3.3 When MPLS is restored:

lab-sw(config)#INT F1/0/44
lab-sw(config-if)#no SHUT

ASA1 partial routing table:

lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
S 192.168.34.4 255.255.255.255 [1/0] via 192.168.123.3, inside
S 192.168.44.0 255.255.255.0 [1/0] via 192.168.123.3, inside
S 192.168.44.44 255.255.255.255 [1/0] via 192.168.123.3, inside
C 192.168.123.0 255.255.255.0 is directly connected, inside
L 192.168.123.1 255.255.255.255 is directly connected, inside

ASA1 tracking object 210 status:

lab-ASA1/pri/act# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
16 changes, last change 00:02:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0

ASA4 routing table

lab-ASA4# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 12.12.12.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 12.12.12.11, internet
S 2.2.2.1 255.255.255.255 [1/0] via 12.12.12.22, internet
C 12.12.12.0 255.255.255.0 is directly connected, internet
L 12.12.12.4 255.255.255.255 is directly connected, internet
S 192.168.0.0 255.255.0.0 [1/0] via 192.168.34.3, c3
C 192.168.34.0 255.255.255.0 is directly connected, c3
L 192.168.34.4 255.255.255.255 is directly connected, c3
C 192.168.44.0 255.255.255.0 is directly connected, production
L 192.168.44.4 255.255.255.255 is directly connected, production
S 192.168.123.1 255.255.255.255 [1/0] via 192.168.34.3, c3

ASA4 tracking object 210 status:

lab-ASA4# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
14 changes, last change 00:03:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0

ASA1 clear IPsec:

lab-ASA1/pri/act# clear crypto ipsec sa peer 12.12.12.4

ASA4 clear IPsec:

lab-ASA4# clear crypto ipsec sa peer 2.2.2.1

ASA3 can ping 192.168.44.44

lab-ASA3# ping 192.168.44.44
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.44.44, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

ASA1 IPSec ISAKMP status:

lab-ASA1/pri/act# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs

ASA1 IPSec IKEV2 status:

lab-ASA1/pri/act# sh crypto ikev2 sa
There are no IKEv2 SAs

ASA4 IPSec ISAKMP status:

lab-ASA4# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs

ASA4 IPSec IKEV2 status:

lab-ASA4# sh crypto ikev2 sa
There are no IKEv2 SAs

Side note #1:
During troubleshooting, I found when ASA3's c3 interface went down, the VPN tunnel was not triggered. And the interesting traffic ACL matched interesting traffic.
All the VPN configurations are correct.
Why the VPN tunnel was not triggered?

I turned on all IPSec debug, but since the VPN is not triggered, no debugging info was shown.

After doing more research, it turns out it's because ASA1 end system is behind NAT. I have use NAT exemption for the VPN traffic.
Once I configured
nat (inside,outside2) source static Office Office destination static production_subnts production_subnts
on ASA1, the VPN went UP immediately.

In the very beginning, I thought NAT-Traversal is the default for IKEV2, so I don't need to use NAT exemption for VPN traffic. But the fact is the NAT-Traversal is within the VPN, VPN cannot be triggered on ASA without NAT exemption. I learned my lesson!!

Now everything works as it should be. ^_^

Last note: I hate static routing!!

INWK05 10週年聚会(2015)

毕业有十年了，同学们决定聚一聚，是啊，人生有多少个10年呢？

大家的样子都没怎么变，挺好。

大家的话题不外乎是工作，赚钱，房子，呵呵。

因为是包房，好像要点够300块的样子，我们竟然点得差不多的样子(稍稍超出了点)。(完)

Pan Am Toronto 2015

2015年的泛美运动会(Pan American Game)在多伦多举行。

本来没觉得有多大热情，可是自从参加Cisco Connect，因为是前400名注册的，Cisco送我两张门票，忽然间就和这运动会有了bond。
再后来，70同学的公司是Pan Am的partner(Cisco 是sponsor)，自然也有送票。70同学还被抽中了。我们家忽然间有了四张门票，一下子就显得富裕起来了！^_^

也因为有了bond， 我们也变得积极起来。开幕式那天，70同学问我要不要去City Hall参加开幕式。我二话没说就答应了。后来才知道，Mississauga的City Hall其实就是放个大屏幕让大家一起看，并没有什么concert和表演，和Toronto DT没法比。

不过我们也说是很积极了，从家里出发，跑到Mississauga City Hall。70同学说，其他人都是走路进场的，我们是跑步进场的，别人还以为我们是运动员呢！LOL

-.跑步进场的“运动员”.-

-.进场前，先去旁边的Library.-

-.其实不是完全没有表演，还是有小丑的.-

加拿大的开幕式一如既往的好，除了有一两个比较松散外，水准还是很高的，我们大家都看得很开心。

-.惊艳.-

-.美美哒.-

-.加拿大队进场.-

传递火炬，有个最惊艳的就是从CN Tower飞下来传递火炬，很有创意呢。
多伦多由于太穷，所以放烟花也不会大规模地放。不过最让我惊喜的，是在CN Tower上放烟花，非常吸睛，很好的创意。

因为太晚了，我们没等结束就回家了。回家继续看电视才发现，Pan Am的开幕式是没有点火炬的，有点小失望。

================================== 分割线 ==================================

我们观看的第一场比赛是女子篮球，是Siemens提供的票。
其实一开始，我都挺失望的——女子篮球有什么好看的。不过觉得难得去观看比赛，好好休息一天也是不错的选择。
谁知道看了以后，才发现国际赛的水平真的高，女篮也非常好看。像看的第一场是Cuba vs Argentina，Cuba竟然是全场紧逼，Argentina被打得捆手捆脚的，不知道有多别扭，快攻是完全打不出来了！

-.selfie.-

-.Siemens的位置空空如也.-

-.由于观众不多，我们找了个更佳的位置.-

我们看了一下这两支队的世界排名，Cuba排13，Argentina排14，旗鼓相当！
最后的结果是68:55，Cuba胜!

由于是初赛，Canada队员也来看Cuba比赛，可惜在我们的对面看台，完全不能去讨个合影什么的。

还有其他球队的探子来做记录，各式其式，百般武艺，各显神通！

-.Pachi出来表演一下.-

第二场是Puerto Rico vs Dominican，最后结果是76:54。Puerto Rico世界排名28，Dominican世界排名37。
虽然两队排名有些差距，可是打起来更好看，互有攻守，节奏比第一场要快，看起来也过瘾些。

我和70同学大呼过瘾，对下周的男篮就更加期待了。

比赛后，才三点多，我们决定去Nathan Fielder Square去凑凑热闹

我们发现Camera man其实超辛苦，整场没有休息；这么重的机器抗在身上几个小时，真的不是开玩笑的。而且整个Pan Am Game都要辛苦工作，我们不喜欢看的比赛可以不看，可是camera man可不能不工作，这两周下来不间断的工作，真得会累到吐血吧！！

后来我们有去了MEC朝拜了一下，还给我买了双backpacking boot，好开森！

-.先去Tim仔补充一下.-


-.我的backpacking boot.-

由于我们早上错过了最后一班Go Train，只能坐地铁了。后来才发现Yorkdale不能停车了，先绕到Downsview发现没停车位了，最后绕到Wilson才有车位，好一番周折。

================================== 分割线 ==================================

由于第一次错过Go Train，我们接下来那周看男篮的比赛，就早早起床，去赶Go Train，这次赶上了！

-.搭火车.-

-.play me!.-

-.Union Station.-

-.山寨帆船酒店.-

-.CN Tower.-

男子比赛就是比较popular，70说他想帮Josie看看还有没有球票，竟然都sold out了。
后来据我们观察，可能卖的票没有了，赠票还是有很多空位置的。

我们看了男篮的世界排名，Brazil排第9，Mexico排19，Dominican排20，Venezuela排27。

第一场Dominican vs Mexico，95:70, Dominican胜。
第二场Venezuela vs Brazil， 64:79，Brazil胜。

-.你滚(巴西球员在做准备运动).-

-.又见camera man.-

我们一张球票，可以看两场比赛，可是不能出去，出去就回不来了。所以只能在球场吃午饭。由于我们怕人多，在第三节就跑去买吃的，所以等大家吃午饭的时候，我们都没事干了。
百无聊赖之际，我们决定买popcorn吃，实在太无聊了呀！

-.popcorn.-

男子比赛明显受欢迎，周边的活动也多起来。有看见Argentina探子来刺探军情，还看见马刺球探来找球员。70说，看见那人拿着马刺队的笔记本，身穿的也是马刺队的队服颜色，应该是马刺的球探：

-.马刺球探.-

说起比赛，怎么会少了各队的fans。

上周有位Puerto Rico 阿姨，那个大嗓门哦。每次她为Puerto Rico高喊加油，70同学都笑到不行，这家伙笑点真低！！

这周呢，有位Dominican 大汉，是位狂热的fans，大喊大叫，比人家队员还大声！后来看见他和队员打招呼，哦，原来有国家队队员做朋友，怪不得这么激动！！
好玩的是，这位Dominican 大汉(和他两位朋友)一起为Dominican加油，旁边有两位Mexico的女士为Mexico加油。不过这两伙人根本不做任何对抗行为，相反互相帮忙：一会儿你帮我拍照，一会儿我帮你看位子，甭提有多和谐，哪里像是为敌队加油的样子啊！！
最好玩的是，中午休息的时候，Dominican的那位大汉，他的国家队队员来和他们打招呼。一开始我们没注意，因为那队员换了便服，我们也不认得他。可是我看见很多人举着相机对着他拍照，再一细看才知道是运动员。当大汉和队员合照完，那位Mexico大妈也过来要跟那队员合照！靠，没搞错吧，你真的是为Mexico加油的吗？！！

巴西的fans就更加多了，而且特别团结，很有组织，气势如虹，Brazil队有福啊！

-.Brazil fans.-

最后值得一提的是，Canada队，世界排名35，竟然半决赛打败了世界第一大美国队，进入决赛！！可惜打决赛对巴西，打得真差，投篮命中率只有22%，偷了别人的球也不能把握机会，篮板也不够巴西，最后输得挺惨的。原来追到只有12分的时候，打了个小高潮，我还以为能还阳了，后来还是没能把握住，好可惜！！

我喜欢看加拿大比赛，无论是冬奥还是夏天奥运会，还是这个Pan Am Game，就是个玩，尽力了，拿到bronze，great! 拿到silver，better！就是要这种全力以赴，尽力了，have fun了，就够了！看加拿大队还是非常舒服和愉快的！(完)

多年后的2nd Cup

感触良多：当初刚来到加拿大读书，省吃俭用，唯一的奢侈就是和70同学去 2nd cup 喝杯和星巴巴一样贵的咖啡。
当时70同学穷得什么似的，却仍然陪我去喝这么贵的咖啡。难怪所有同学都觉得我们是一对，可惜我们真的不是，当时！[害羞]
今天再来喝咖啡，还可以吃块蛋糕，好幸福！[愉快][愉快][愉快]

route leaking between different vrf (using static routing)

I haven't used VRF with static route before. All I used before was vrf with BGP/OPSF/EIGRP.

Here are the solution I found in order to leak traffic between different vrfs using static route.

The BGP on R1 does NOT require BGP neighbor. Also, the vrf RED and BLUE on R1, does NOT require to import its own rd, which is the only difference compared to the solution provided here.
R2 has normal routing without vrf involved.

Configuration

R1(config)#
ip vrf RED
rd 65535:3
route-target export 3:3
route-target import 4:4
!
ip vrf BLUE
rd 65535:4
route-target export 4:4
route-target import 3:3
!
ip route vrf RED 192.168.3.0 255.255.255.0 192.168.1.2
ip route vrf BLUE 192.168.4.0 255.255.255.0 192.168.2.2
!
router bgp 65535
address-family ipv4 vrf RED
redistribute connected
redistribute static
!
address-family ipv4 vrf BLUE
redistribute connected
redistribute static
!

Verification

R1#sh ip bgp vpnv4 all
BGP table version is 13, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 3:3 (default for vrf RED)
*> 192.168.1.0 0.0.0.0 0 32768 ?
*> 192.168.2.0 0.0.0.0 0 32768 ?
*> 192.168.3.0 192.168.1.2 0 32768 i
*> 192.168.4.0 192.168.2.2 0 32768 ?
Route Distinguisher: 4:4 (default for vrf BLUE)
*> 192.168.1.0 0.0.0.0 0 32768 ?
*> 192.168.2.0 0.0.0.0 0 32768 ?
*> 192.168.3.0 192.168.1.2 0 32768 i
*> 192.168.4.0 192.168.2.2 0 32768 ?

R1#ping vrf RED 192.168.4.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/37/60 ms

R1#ping vrf BLUE 192.168.3.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/30/68 ms

*************************************************************************************

After doing more research, there are another way to force VRF leak.

By default, the VRF leak is enabled, which is controlled by "ip route static inter-vrf".

For example, on R1, the complete configuration is:

Configuration

ip vrf RED
rd 65535:3
!
ip route vrf RED 192.168.3.0 255.255.255.0 192.168.1.2
ip route vrf RED 192.168.4.0 255.255.255.0 f0/1 192.168.2.2 global
ip route 192.168.4.0 255.255.255.0 192.168.2.2
ip route 192.168.3.0 255.255.255.0 f0/0 192.168.1.2
!

Verification

R1#sh ip route vrf RED
Routing Table: RED
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set

C 192.168.1.0/24 is directly connected, FastEthernet0/0
S 192.168.3.0/24 [1/0] via 192.168.1.2
S 192.168.4.0/24 [1/0] via 192.168.2.2, FastEthernet0/1

R1#ping vrf RED 192.168.4.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/37/60 ms

R1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set

C 192.168.2.0/24 is directly connected, FastEthernet0/1
S 192.168.3.0/24 [1/0] via 192.168.1.2, FastEthernet0/0
S 192.168.4.0/24 [1/0] via 192.168.2.2

R1#ping 192.168.3.2 source 192.168.2.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/30/68 ms