2015年11月22日星期日

Dual ISP and VPN as MPLS back

Working on a task:

1. Use 2 ISPs as Internet connections, one as active and the other as backup.
2. Use VPN using backup ISP as the alternative way when MPLS circuit down.

Did some research and found useful articles:

ASA/PIX 7.x: Redundant or Backup ISP Links Configuration Example
Configuring LAN-to-LAN VPNs
Ken Felix Security Blog -- Cisco ASA ikev2 setup

Setup a lab to simulate these 2 scenarios:

-.logical connection.-


-.physical connection.-

1. Dual ISP portion
The challenge I faced is not when the primary Internet circuit down, default route using backup ISP router SW2. Track object work perfectly.
The real challenge is when the primary circuit restored, because the default route is still pointing to backup ISP router SW2, the tracking object will not be reachable via the ASA interface (in this example is the outside1 interface). This is the way how ASA works.

The trick is to configure a host route of ping target via the primary interface. No matter whether the primary circuit is down, the ASA1 will always use the host route to ping the ping target.

Here are the configuration:

1.1 ASA1
In this example, I actually use 12.12.12.22 as my ping target (that is because I use 12.12.12.4 in the next scenario)
sla monitor 100
type echo protocol ipIcmpEcho 12.12.12.22 interface outside1
num-packets 3
frequency 10
sla monitor schedule 100 life forever start-time now
!
track 100 rtr 100 reachability
This the default route using primary ISP router SW1
route outside1 0.0.0.0 0.0.0.0 1.1.1.11 1 track 100
!This the 2nd default route using secondary ISP router SW2
route outside2 0.0.0.0 0.0.0.0 2.2.2.22 50
In this example, I actually use 12.12.12.22 as my ping target, so configure the following host route to force ASA1 use outside1 to reach the ping target 12.12.12.22
route outside1 12.12.12.22 255.255.255.255 1.1.1.11 1
NAT configuration:
object network office_subnets1
nat (inside,outside1) dynamic interface
object network office_subnets2
nat (inside,outside2) dynamic interface
1.2 SW1
ip route 0.0.0.0 0.0.0.0 12.12.12.4
1.3 SW2
In this lab environment, I have to also force secondary ISP router SW2 to use SW1 to reach 1.1.1.0/24 network.
ip route 0.0.0.0 0.0.0.0 12.12.12.4
ip route 1.1.1.0 255.255.255.0 12.12.12.11
1.4 Validation

1.4.1 Both primary and secondary Internet circuit are UP:
ASA1's partial routing table:
lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1
ASA1 tracking object 12.12.12.22 status:

lab-ASA1/pri/act# sh track 100
Track 100
Response Time Reporter 100 reachability
Reachability is Up
22 changes, last change 00:00:03
Latest operation return code: OK
Latest RTT (millisecs) 4
Tracked by:
STATIC-IP-ROUTING 0
ASA3 can ping 12.12.12.4:
lab-ASA3# PING 12.12.12.4 rep 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 12.12.12.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1.4.2 When primary Internet circuit is down:
lab-sw(config)#int f1/0/33
lab-sw(config-if)#shut
ASA1 partial routing table:
lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 2.2.2.22 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [50/0] via 2.2.2.22, outside2
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1
ASA1 tracking object 12.12.12.22 status:
lab-ASA1/pri/act# sh track 100
Track 100
Response Time Reporter 100 reachability
Reachability is Down
21 changes, last change 00:00:27
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0
ASA3 can ping 12.12.12.4:
lab-ASA3# PING 12.12.12.4 rep 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 12.12.12.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

1.4.3 When primary Internet circuit restore:
lab-sw(config)#int f1/0/33
lab-sw(config-if)#no shut
ASA1 tracking object 12.12.12.22 status:
lab-ASA1/pri/act# sh track 100
Track 100
Response Time Reporter 100 reachability
Reachability is Up
22 changes, last change 00:00:03
Latest operation return code: OK
Latest RTT (millisecs) 4
Tracked by:
STATIC-IP-ROUTING 0
ASA1's partial routing table:
lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1
ASA3 can ping 12.12.12.4:
lab-ASA3# PING 12.12.12.4 rep 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 12.12.12.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Side note #1:
All the ASA in this lab are running version 9.2(2)4.
The tracking object feature is very limited.
Each track can only include one sla.
It will be ideal if each track can include multiple sla. I will like to have multiple ping targets for my Internet circuit. Rely on one ping target can be very risky.

Side note #2:
If configure:
route outside1 0.0.0.0 0.0.0.0 1.1.1.1 1 track 1
route outside1 0.0.0.0 0.0.0.0 1.1.1.1 2 track 2
This is a "OR" operation.
When track 1 is down, the 2nd route will be installed;
When track 2 is down, the 1st route will be installed;
When both track 1 and 2 are down, the default route will be withdrew.

2. VPN as MPLS backup portion

2.1 ASA1
ASA1 tracking object 210:
sla monitor 210
type echo protocol ipIcmpEcho 192.168.44.44 interface inside
num-packets 3
frequency 10
sla monitor schedule 210 life forever start-time now
!
track 210 rtr 210 reachability
Static routes:
route inside 192.168.44.0 255.255.255.0 192.168.123.3 1 track 210
! host route for ping target
route inside 192.168.34.4 255.255.255.255 192.168.123.3 1
VPN configuration:
crypto ipsec ikev2 ipsec-proposal ESP-AES-SHA512
protocol esp encryption aes
protocol esp integrity sha-512
!
crypto map L2LVPN 10 match address acl_office_to_prod_vpn
crypto map L2LVPN 10 set peer 12.12.12.4
crypto map L2LVPN 10 set ikev2 ipsec-proposal ESP-AES-SHA512
crypto map L2LVPN interface outside2
!
crypto isakmp identity address
!
crypto ikev2 policy 10
encryption aes
integrity sha512
group 2
prf sha
lifetime seconds 28800
crypto ikev2 enable outside2
!
tunnel-group 12.12.12.4 type ipsec-l2l
tunnel-group 12.12.12.4 ipsec-attributes
ikev2 remote-authentication pre-shared-key password
ikev2 local-authentication pre-shared-key password
!
object-group network Office
network-object 192.168.123.0 255.255.255.0
!
object-group network production_subnts
network-object 192.168.44.0 255.255.255.0
!
access-list acl_office_to_prod_vpn extended permit ip object-group Office object-group production_subnts
!
nat (inside,outside2) source static Office Office destination static production_subnts production_subnts
2.2 ASA4
ASA4 tracking object 210:
sla monitor 210
type echo protocol ipIcmpEcho 192.168.123.1 interface c3
num-packets 3
frequency 10
sla monitor schedule 210 life forever start-time now
!
track 210 rtr 210 reachability
Static routes:
route c3 192.168.0.0 255.255.0.0 192.168.34.3 1 track 210
route c3 192.168.0.0 255.255.0.0 12.12.12.22 50

route internet 0.0.0.0 0.0.0.0 12.12.12.11 1
! host route for the VPN end system, make it use ASA1's outside2 interface, not ASA1's outside1
route internet 2.2.2.1 255.255.255.255 12.12.12.22 1
! host route for ping target
route c3 192.168.123.1 255.255.255.255 192.168.34.3 1
VPN configuration:
crypto ipsec ikev2 ipsec-proposal ESP-AES-SHA512
protocol esp encryption aes
protocol esp integrity sha-512
!
crypto map L2LVPN 10 match address acl_prod_to_office_vpn
crypto map L2LVPN 10 set peer 2.2.2.1
crypto map L2LVPN 10 set ikev2 ipsec-proposal ESP-AES-SHA512
crypto map L2LVPN interface internet
!
crypto isakmp identity address
!
crypto ikev2 policy 10
encryption aes
integrity sha512
group 2
prf sha
lifetime seconds 28800
crypto ikev2 enable internet
!
tunnel-group 2.2.2.1 type ipsec-l2l
tunnel-group 2.2.2.1 ipsec-attributes

ikev2 remote-authentication pre-shared-key password
ikev2 local-authentication pre-shared-key password
!
object-group network Office
network-object 192.168.123.0 255.255.255.0
!
object-group network production_subnts
network-object 192.168.44.0 255.255.255.0
!
access-list acl_prod_to_office_vpn extended permit ip object-group Office object-group production_subnts
!

2.3 Validation

2.3.1 When MPLS is UP:
ASA1 partial routing table:

lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
S 192.168.34.4 255.255.255.255 [1/0] via 192.168.123.3, inside
S 192.168.44.0 255.255.255.0 [1/0] via 192.168.123.3, inside
S 192.168.44.44 255.255.255.255 [1/0] via 192.168.123.3, inside
C 192.168.123.0 255.255.255.0 is directly connected, inside
L 192.168.123.1 255.255.255.255 is directly connected, inside
ASA1 tracking object 210 status:

lab-ASA1/pri/act# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
16 changes, last change 00:02:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0
ASA1 IPSec ISAKMP status:
lab-ASA1/pri/act# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs

ASA1 IPSec IKEV2 status:
lab-ASA1/pri/act# sh crypto ikev2 sa
There are no IKEv2 SAs
ASA4 routing table
lab-ASA4# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 12.12.12.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 12.12.12.11, internet
S 2.2.2.1 255.255.255.255 [1/0] via 12.12.12.22, internet
C 12.12.12.0 255.255.255.0 is directly connected, internet
L 12.12.12.4 255.255.255.255 is directly connected, internet
S 192.168.0.0 255.255.0.0 [1/0] via 192.168.34.3, c3
C 192.168.34.0 255.255.255.0 is directly connected, c3
L 192.168.34.4 255.255.255.255 is directly connected, c3
C 192.168.44.0 255.255.255.0 is directly connected, production
L 192.168.44.4 255.255.255.255 is directly connected, production
S 192.168.123.1 255.255.255.255 [1/0] via 192.168.34.3, c3
ASA4 tracking object 210 status:

lab-ASA4# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
14 changes, last change 00:03:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0
ASA4 IPSec ISAKMP status:
lab-ASA4# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs
ASA4 IPSec IKEV2 status:
lab-ASA4# sh crypto ikev2 sa
There are no IKEv2 SAs
ASA3 can ping 192.168.44.44
lab-ASA3# ping 192.168.44.44
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.44.44, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

2.3.2 When MPLS is DOWN:
lab-sw(config)#INT F1/0/44
lab-sw(config-if)#SHUT
ASA1 partial routing table:
lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
C 1.1.1.0 255.255.255.0 is directly connected, outside1
L 1.1.1.1 255.255.255.255 is directly connected, outside1
C 2.2.2.0 255.255.255.0 is directly connected, outside2
L 2.2.2.1 255.255.255.255 is directly connected, outside2
S 12.12.12.4 255.255.255.255 [1/0] via 2.2.2.22, outside2
S 12.12.12.22 255.255.255.255 [1/0] via 1.1.1.11, outside1
C 172.16.0.0 255.255.255.252 is directly connected, failover
L 172.16.0.1 255.255.255.255 is directly connected, failover
S 192.168.34.4 255.255.255.255 [1/0] via 192.168.123.3, inside
S 192.168.44.0 255.255.255.0 [50/0] via 2.2.2.22, outside2
S 192.168.44.44 255.255.255.255 [1/0] via 192.168.123.3, inside
C 192.168.123.0 255.255.255.0 is directly connected, inside
L 192.168.123.1 255.255.255.255 is directly connected, inside
ASA1 tracking object 210
lab-ASA1/pri/act# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Down
17 changes, last change 00:00:00
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0
ASA4 routing table:
lab-ASA4# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 12.12.12.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 12.12.12.11, internet
S 2.2.2.1 255.255.255.255 [1/0] via 12.12.12.22, internet
C 12.12.12.0 255.255.255.0 is directly connected, internet
L 12.12.12.4 255.255.255.255 is directly connected, internet
S 192.168.0.0 255.255.0.0 [50/0] via 12.12.12.22, c3
C 192.168.34.0 255.255.255.0 is directly connected, c3
L 192.168.34.4 255.255.255.255 is directly connected, c3
C 192.168.44.0 255.255.255.0 is directly connected, production
L 192.168.44.4 255.255.255.255 is directly connected, production
S 192.168.123.1 255.255.255.255 [1/0] via 192.168.34.3, c3
ASA3 tracking object 210
lab-ASA4# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Down
15 changes, last change 00:00:38
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0
ASA3 can ping 192.168.44.44:
lab-ASA3# ping 192.168.44.44
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.44.44, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms
ASA1 IPsec ISAKMP status:
lab-ASA1/pri/act# sh crypto isakmp sa
There are no IKEv1 SAs
IKEv2 SAs:
Session-id:6, Status:UP-ACTIVE, IKE count:1, CHILD count:1
Tunnel-id Local Remote Status Role
219900601 2.2.2.1/500 12.12.12.4/500 READY INITIATOR
Encr: AES-CBC, keysize: 128, Hash: SHA512, DH Grp:2, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/147 sec
Child sa: local selector 192.168.123.0/0 - 192.168.123.255/65535
remote selector 192.168.44.0/0 - 192.168.44.255/65535
ESP spi in/out: 0x656293c/0x2a61c4bb 
ASA1 IPsec IKEV2 status:
lab-ASA1/pri/act# sh crypto ikev2 sa
IKEv2 SAs:
Session-id:6, Status:UP-ACTIVE, IKE count:1, CHILD count:1
Tunnel-id Local Remote Status Role
219900601 2.2.2.1/500 12.12.12.4/500 READY INITIATOR
Encr: AES-CBC, keysize: 128, Hash: SHA512, DH Grp:2, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/140 sec
Child sa: local selector 192.168.123.0/0 - 192.168.123.255/65535
remote selector 192.168.44.0/0 - 192.168.44.255/65535
ESP spi in/out: 0x656293c/0x2a61c4bb 
2.3.3 When MPLS is restored:
lab-sw(config)#INT F1/0/44
lab-sw(config-if)#no SHUT
ASA1 partial routing table:
lab-ASA1/pri/act# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.1.1.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 1.1.1.11, outside1
S 192.168.34.4 255.255.255.255 [1/0] via 192.168.123.3, inside
S 192.168.44.0 255.255.255.0 [1/0] via 192.168.123.3, inside
S 192.168.44.44 255.255.255.255 [1/0] via 192.168.123.3, inside
C 192.168.123.0 255.255.255.0 is directly connected, inside
L 192.168.123.1 255.255.255.255 is directly connected, inside
ASA1 tracking object 210 status:
lab-ASA1/pri/act# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
16 changes, last change 00:02:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0
ASA4 routing table
lab-ASA4# sh route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 12.12.12.11 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 12.12.12.11, internet
S 2.2.2.1 255.255.255.255 [1/0] via 12.12.12.22, internet
C 12.12.12.0 255.255.255.0 is directly connected, internet
L 12.12.12.4 255.255.255.255 is directly connected, internet
S 192.168.0.0 255.255.0.0 [1/0] via 192.168.34.3, c3
C 192.168.34.0 255.255.255.0 is directly connected, c3
L 192.168.34.4 255.255.255.255 is directly connected, c3
C 192.168.44.0 255.255.255.0 is directly connected, production
L 192.168.44.4 255.255.255.255 is directly connected, production
S 192.168.123.1 255.255.255.255 [1/0] via 192.168.34.3, c3
ASA4 tracking object 210 status:
lab-ASA4# sh track 210
Track 210
Response Time Reporter 210 reachability
Reachability is Up
14 changes, last change 00:03:42
Latest operation return code: OK
Latest RTT (millisecs) 1
Tracked by:
STATIC-IP-ROUTING 0
ASA1 clear IPsec:
lab-ASA1/pri/act# clear crypto ipsec sa peer 12.12.12.4
ASA4 clear IPsec:
lab-ASA4# clear crypto ipsec sa peer 2.2.2.1
ASA3 can ping 192.168.44.44
lab-ASA3# ping 192.168.44.44
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.44.44, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

ASA1 IPSec ISAKMP status:
lab-ASA1/pri/act# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs
ASA1 IPSec IKEV2 status:
lab-ASA1/pri/act# sh crypto ikev2 sa
There are no IKEv2 SAs
ASA4 IPSec ISAKMP status:
lab-ASA4# sh crypto isakmp sa
There are no IKEv1 SAs
There are no IKEv2 SAs
ASA4 IPSec IKEV2 status:
lab-ASA4# sh crypto ikev2 sa
There are no IKEv2 SAs
Side note #1:
During troubleshooting, I found when ASA3's c3 interface went down, the VPN tunnel was not triggered. And the interesting traffic ACL matched interesting traffic.
All the VPN configurations are correct.
Why the VPN tunnel was not triggered?

I turned on all IPSec debug, but since the VPN is not triggered, no debugging info was shown.

After doing more research, it turns out it's because ASA1 end system is behind NAT. I have use NAT exemption for the VPN traffic.
Once I configured
nat (inside,outside2) source static Office Office destination static production_subnts production_subnts
on ASA1, the VPN went UP immediately.

In the very beginning, I thought NAT-Traversal is the default for IKEV2, so I don't need to use NAT exemption for VPN traffic. But the fact is the NAT-Traversal is within the VPN, VPN cannot be triggered on ASA without NAT exemption. I learned my lesson!!

Now everything works as it should be. ^_^

Last note: I hate static routing!!