IPsec – ESP, AH, IKE Phases, Tunnel Mode & VPN Configuration

1. What Is IPsec?

IPsec (Internet Protocol Security) is an open-standard framework of protocols that secures IP communications by providing authentication, integrity, and encryption at the network layer (Layer 3). Unlike application-layer security (TLS/HTTPS), IPsec protects all traffic regardless of which application generates it — making it transparent to applications and ideal for building VPNs (Virtual Private Networks).

IPsec operates between two peers — routers, firewalls, or hosts — and protects the IP packets flowing between them. Before any protected traffic flows, the peers must negotiate a set of security parameters and establish Security Associations (SAs) using the IKE (Internet Key Exchange) protocol.

  Without IPsec (plaintext across the internet):
  [PC at HQ] ──── Internet ────────────────────── [Server at Branch]
              Anyone on path can read/modify packets

  With IPsec site-to-site tunnel:
  [PC at HQ]──[Router A]═══[ENCRYPTED TUNNEL]═══[Router B]──[Server at Branch]
                192.168.1.0/24      Internet       192.168.2.0/24
  Traffic is encrypted between Router A and Router B;
  PCs and servers need no special configuration

IPsec delivers three core security services:

Service	Mechanism	What It Prevents
Confidentiality	Symmetric encryption (AES, 3DES)	Eavesdropping — attacker cannot read intercepted packets
Integrity	HMAC (SHA-256, SHA-384)	Tampering — any modification to a packet is detected and the packet is discarded
Authentication	Pre-shared keys or digital certificates	Impersonation — verifies the remote peer is who it claims to be
Anti-Replay	Sequence numbers in ESP/AH headers	Replay attacks — attacker cannot capture and re-send valid encrypted packets to trick the receiver

2. IPsec Protocol Components: AH and ESP

IPsec uses two protocol headers to secure packets. These are applied to the original IP packet either in Transport mode or Tunnel mode (covered in Section 3).

Authentication Header (AH) — Protocol 51

AH provides integrity, source authentication, and anti-replay protection for the entire IP packet — including the original IP header. It does not provide encryption. AH is identified by IP Protocol Number 51.

  AH in Transport Mode:
  ┌───────────┬──────────┬─────────────────┐
  │ IP Header │ AH Header│ Original Payload │
  └───────────┴──────────┴─────────────────┘
               ↑ Integrity check covers HMAC over IP header + payload
               ↑ No encryption — payload is still readable

Encapsulating Security Payload (ESP) — Protocol 50

ESP provides confidentiality (encryption), integrity, source authentication, and anti-replay. It is the most widely used IPsec component and is standard in all modern VPN deployments. ESP is identified by IP Protocol Number 50.

  ESP in Tunnel Mode (site-to-site VPN — most common):
  ┌──────────────┬────────────┬────────────────┬─────────────────────────────┬───────────┬──────────┐
  │ New IP Header│ ESP Header │ Original IP Hdr│ Original Payload            │ ESP Trailer│ ESP Auth │
  └──────────────┴────────────┴─────────────────────────────────────────────┴───────────┴──────────┘
                               ←─────────────────── ENCRYPTED ──────────────────────────→
                                         ←────────────── INTEGRITY CHECK ───────────────────→

  ESP Header contains: SPI (32 bits) | Sequence Number (32 bits)
  ESP Trailer contains: Padding | Pad Length | Next Header
  ESP Auth: HMAC value over ESP Header + Encrypted data + ESP Trailer

AH vs ESP — Full Comparison

Feature	AH (Protocol 51)	ESP (Protocol 50)
Encryption (Confidentiality)	No — payload remains readable	Yes — payload is encrypted
Integrity	Yes — covers entire packet including IP header	Yes — covers ESP header + payload + trailer
Source Authentication	Yes	Yes (optional — ESP-auth)
Anti-Replay	Yes	Yes
IP header integrity	Yes — covers original IP header fields	No — new outer IP header not covered by integrity
NAT compatibility	Breaks with NAT — NAT changes IP header, invalidating AH integrity check	Compatible with NAT-T (UDP encapsulation)
IP Protocol Number	51	50
Deployment	Rare — legacy, used when encryption prohibited by policy but authentication required	Universal — standard for all VPN deployments

AH and NAT are fundamentally incompatible. AH calculates an integrity value over the source IP address in the original IP header. When a NAT device translates the source IP, it changes the field that AH has already signed — the receiving end's integrity check fails and the packet is dropped. This is why AH is almost never used in real-world deployments where NAT is present (i.e., virtually everywhere). ESP with NAT-T is the universal solution.

3. IPsec Modes: Transport vs Tunnel

IPsec can operate in two modes that determine which part of the original packet is protected and whether a new IP header is added.

Transport Mode

  Original packet:
  ┌───────────┬────────────────────────────────┐
  │ IP Header │ TCP/UDP Header + Payload        │
  │ (src/dst) │                                 │
  └───────────┴────────────────────────────────┘

  After ESP Transport Mode:
  ┌───────────┬────────────┬────────────────────────────────┬───────────┬──────────┐
  │ IP Header │ ESP Header │ TCP/UDP Header + Payload (ENC) │ ESP Trail │ ESP Auth │
  │(unchanged)│            │                                │           │          │
  └───────────┴────────────┴────────────────────────────────┴───────────┴──────────┘

  The original IP header is preserved — src/dst IPs are the actual host IPs
  Only the upper-layer payload (TCP/UDP + data) is encrypted

What is protected: Upper-layer payload (TCP/UDP header + data) only
IP header: Original IP header unchanged — source and destination IPs are visible to anyone on the path
Use case: Host-to-host communication where both endpoints are IPsec-capable (e.g., server-to-server database replication, Microsoft L2TP/IPsec client VPN inner encryption)
Overhead: Lower — no new IP header added

Tunnel Mode

  Original packet:
  ┌───────────┬────────────────────────────────┐
  │ IP Header │ TCP/UDP Header + Payload        │
  │192.168.1.10→192.168.2.10                   │
  └───────────┴────────────────────────────────┘

  After ESP Tunnel Mode:
  ┌──────────────┬────────────┬──────────────────────────────────────────┬──────────┐
  │ New IP Header│ ESP Header │ [Original IP Hdr + TCP/UDP + Payload ENC]│ ESP Auth │
  │198.51.100.1 → 203.0.113.1│                                           │          │
  └──────────────┴────────────┴──────────────────────────────────────────┴──────────┘
  ↑ Outer header = VPN gateway IPs (visible)
                               ↑ Entire original packet = encrypted (hidden inside tunnel)

What is protected: Entire original IP packet — header and payload
IP header: New outer IP header added with VPN gateway addresses; original src/dst IPs are hidden inside the encrypted payload
Use case: Site-to-site VPNs (gateway-to-gateway), remote-access VPNs — the default and overwhelmingly dominant IPsec mode
Overhead: Higher — new IP header + ESP headers add ~50–60 bytes per packet

Feature	Transport Mode	Tunnel Mode
What is encrypted	Payload only (TCP/UDP + data)	Entire original packet (IP header + payload)
Original IP header visible	Yes — src/dst IPs of actual hosts exposed	No — hidden inside encrypted envelope
New IP header added	No	Yes — outer header with VPN gateway IPs
Typical use	Host-to-host (both ends IPsec-aware)	Site-to-site VPN, remote-access VPN (default)
Header overhead	Lower	Higher (~20 bytes extra for new IP header)

4. Security Associations (SA) and the SAD

A Security Association (SA) is a one-directional logical relationship between two IPsec peers that defines all the parameters needed to protect traffic in that direction. Because an SA is unidirectional, two SAs are needed for bidirectional communication (one for each direction).

Each SA contains:

SPI (Security Parameter Index): A 32-bit value in the ESP/AH header that uniquely identifies the SA. The receiving peer uses the SPI to look up the correct SA in the Security Association Database (SAD).
Destination IP: The endpoint this SA applies to
IPsec protocol: AH or ESP
Encryption algorithm and key: e.g., AES-256 + 256-bit key
Integrity algorithm and key: e.g., HMAC-SHA-256 + key
Mode: Transport or Tunnel
SA lifetime: In seconds or bytes — SA is re-keyed when lifetime expires
Sequence number: Current anti-replay counter

  For an IPsec tunnel between Router A and Router B:

  SA #1 (Inbound to Router A, Outbound from Router B):
  SPI=0xABC123, ESP, AES-256, HMAC-SHA256, Tunnel mode, lifetime=3600s

  SA #2 (Outbound from Router A, Inbound to Router B):
  SPI=0xDEF456, ESP, AES-256, HMAC-SHA256, Tunnel mode, lifetime=3600s

  Each SA is stored in the Security Association Database (SAD)
  Router A's SAD: Two entries — one inbound, one outbound
  Router B's SAD: Two entries — mirror of Router A's SAD

5. IKE — Internet Key Exchange

IKE (Internet Key Exchange) is the protocol that automatically negotiates and manages IPsec Security Associations. Without IKE, both peers would need to manually configure identical encryption keys — unscalable and insecure for large networks. IKE runs over UDP port 500 (and UDP 4500 when NAT-T is active).

IKE operates in two phases:

IKE Phase 1 — Building the Management Channel (ISAKMP SA)

Phase 1 establishes a secure, authenticated channel between the two peers — the ISAKMP SA (Internet Security Association and Key Management Protocol). This management channel is then used to safely negotiate the actual IPsec SAs in Phase 2.

  Phase 1 negotiation (IKEv1 Main Mode — 6 messages):

  Peer A                                    Peer B
    │── Message 1: Propose ISAKMP policy ──▶  │  (encryption, hash, auth, DH group, lifetime)
    │◀─ Message 2: Accept/Reject policy ──    │
    │── Message 3: DH public key + nonce ──▶  │  (start Diffie-Hellman exchange)
    │◀─ Message 4: DH public key + nonce ──   │
    │                                         │
    [Both sides independently compute shared secret using DH]
    [Session keys derived from: DH secret + nonces]
    │                                         │
    │── Message 5: Auth payload (encrypted) ▶ │  (pre-shared key or digital cert)
    │◀─ Message 6: Auth payload (encrypted) ─ │
    │                                         │
    ISAKMP SA established (Phase 1 complete)
    Purpose: Protects Phase 2 negotiation
    Lifetime: Typically 86400 seconds (24 hours)

Phase 1 parameters negotiated (must match exactly on both peers):

Parameter	What It Defines	Recommended Value	Legacy (Avoid)
Encryption	Algorithm for protecting Phase 1 messages	AES-256	DES, 3DES
Hash / Integrity	HMAC algorithm for message integrity	SHA-256, SHA-384	MD5, SHA-1
Authentication	How peers prove identity	Digital certificates (RSA/ECDSA)	Pre-shared key (PSK) — simpler but less scalable
DH Group	Diffie-Hellman group for key exchange	Group 14 (2048-bit) or higher; Groups 19/20 (ECDH)	Group 1 (768-bit), Group 2 (1024-bit) — deprecated
Lifetime	How long Phase 1 SA is valid before re-keying	86400 seconds (24 hours)	Very short lifetimes increase re-keying overhead

IKE Phase 2 — Negotiating the IPsec SAs (Quick Mode)

Phase 2 uses the secure channel established in Phase 1 to negotiate the actual IPsec SAs that will protect the data traffic. This is called Quick Mode in IKEv1 and completes in 3 messages.

  Phase 2 (Quick Mode — 3 messages, protected by Phase 1 ISAKMP SA):

  Peer A                                    Peer B
    │── Propose IPsec SA parameters ───────▶  │  (ESP/AH, encryption, hash, lifetime,
    │                                         │   traffic selectors/ACL, PFS DH group)
    │◀─ Accept parameters ────────────────    │
    │── Confirm ──────────────────────────▶   │
    │                                         │
    IPsec SAs established (two — one per direction)
    Purpose: Protects actual user data traffic
    Lifetime: Typically 3600 seconds (1 hour) and/or data volume limit

Phase 2 parameters negotiated:

Parameter	What It Defines	Example
IPsec Protocol	AH or ESP	ESP (virtually universal)
Encryption	Algorithm for encrypting user data	AES-256-GCM (authenticated encryption — no separate hash needed)
Integrity (Hash)	HMAC for data integrity	SHA-256 (with non-GCM cipher suites)
Mode	Transport or Tunnel	Tunnel (site-to-site VPN default)
Traffic Selectors	Which traffic to encrypt (ACL)	192.168.1.0/24 ↔ 192.168.2.0/24
PFS (DH Group)	New DH exchange for each Phase 2 SA (optional)	Group 14 or Group 20 (ECDH)
Lifetime	How long IPsec SAs are valid	3600 seconds (1 hour) — shorter than Phase 1

6. IKEv1 vs IKEv2

Feature	IKEv1	IKEv2
RFC	RFC 2409 (1998)	RFC 7296 (2014)
Phase 1 messages	6 (Main Mode) or 3 (Aggressive Mode)	4 (combined into single exchange — IKE_SA_INIT + IKE_AUTH)
Message exchanges	Phase 1 + Phase 2 (separate, sequential)	Single unified exchange — faster and simpler
NAT Traversal	Requires separate NAT-T extension (RFC 3947)	Built-in — automatic detection and handling
EAP authentication	Not supported	Supported — enables remote-access VPN with username/password
Dead Peer Detection (DPD)	Extension (RFC 3706)	Built-in liveness check
MOBIKE (mobility)	Not supported	Supported — VPN survives IP address change (mobile clients)
Cisco IOS command	`crypto isakmp policy`	`crypto ikev2 proposal`
Recommended?	Legacy — still widely deployed	Yes — preferred for all new deployments

7. Diffie-Hellman Key Exchange and Perfect Forward Secrecy

Diffie-Hellman (DH) Key Exchange

Diffie-Hellman is the mathematical mechanism that allows two peers to independently derive the same shared secret key over an untrusted channel — without ever transmitting the key itself. IPsec uses DH during Phase 1 (and optionally Phase 2 with PFS) to generate the keying material.

  Diffie-Hellman — simplified:

  Peer A and Peer B agree publicly on DH group parameters (prime p, generator g)

  Peer A:  generates secret a → computes public value A = g^a mod p → sends A
  Peer B:  generates secret b → computes public value B = g^b mod p → sends B

  Peer A:  computes shared secret = B^a mod p = g^(ab) mod p
  Peer B:  computes shared secret = A^b mod p = g^(ab) mod p

  Both arrive at the same shared secret without ever transmitting it
  An interceptor who captured A and B cannot compute g^(ab) mod p
  without solving the Discrete Logarithm Problem (computationally infeasible
  with large groups)

DH Group Recommendations

DH Group	Key Size / Type	Status	Recommendation
Group 1	768-bit MODP	Deprecated	Never use — trivially broken
Group 2	1024-bit MODP	Deprecated	Avoid — vulnerable to nation-state attacks; still common in legacy config
Group 5	1536-bit MODP	Deprecated	Avoid — marginal improvement over Group 2
Group 14	2048-bit MODP	Acceptable	Minimum acceptable for new deployments; widely supported
Group 19	256-bit ECDH (P-256)	Recommended	Smaller key, same security as 3072-bit MODP; efficient
Group 20	384-bit ECDH (P-384)	Recommended	Higher security; used in NSA Suite B / CNSA compliance

Perfect Forward Secrecy (PFS)

PFS ensures that if a long-term key (the Phase 1 Diffie-Hellman key) is ever compromised, past Phase 2 session keys cannot be derived from it. Without PFS, all Phase 2 SAs use keying material derived from the Phase 1 DH exchange — compromise of Phase 1 material could retroactively decrypt past IPsec traffic.

With PFS enabled, each Phase 2 negotiation performs a fresh independent DH exchange. Each session's encryption keys are derived from this new, ephemeral DH secret. Compromising one session's keys reveals nothing about any other session.

PFS trade-off: PFS increases CPU overhead during Phase 2 re-keying because a new DH computation is performed each time (Phase 2 lifetime is typically 1 hour, so this happens frequently). For high-security environments the trade-off is worthwhile; for lower-security environments with high tunnel throughput, it may be disabled for performance reasons.

8. NAT Traversal (NAT-T)

NAT-T (NAT Traversal) solves the fundamental incompatibility between IPsec ESP and NAT devices. The problem: ESP is a Layer 3 protocol — it has no port numbers. A NAT device that performs PAT (Port Address Translation) works by tracking connections using source port numbers. Since ESP has no ports, PAT-based NAT cannot correctly demultiplex ESP packets to the right internal host. Additionally, NAT modifies the outer IP source address, which can interfere with IPsec integrity checking.

  Problem without NAT-T:

  Internal PC ──── Router A (behind NAT) ──── Internet ──── Router B
  192.168.1.1      10.0.0.1/1.2.3.4                          5.6.7.8

  Router A sends ESP packet: src=10.0.0.1, proto=50 (ESP)
  NAT rewrites: src=1.2.3.4, proto=50 (ESP)
  Router B receives: src=1.2.3.4, proto=50 → processes fine (outbound OK)

  Router B sends ESP reply: dst=1.2.3.4, proto=50
  NAT at Router A sees: dst=1.2.3.4, proto=50 (no port!)
  NAT cannot determine which internal host this belongs to → DROPS packet

  Solution — NAT-T:
  ┌─────────────────────────────────────────────────────────────────┐
  │ Outer IP Header │ UDP Header (src/dst port 4500) │ ESP Packet │
  └─────────────────────────────────────────────────────────────────┘
  NAT can now track by UDP port 4500 → correctly demultiplexes
  Both peers detected NAT during IKE negotiation (IKE NAT-D payloads)
  and automatically switched to UDP 4500 encapsulation

Key NAT-T details:

IKE negotiation still starts on UDP 500 — NAT is detected during Phase 1 using NAT-D (NAT Discovery) payloads
If NAT is detected, both peers switch to UDP 4500 for the remainder of IKE and all ESP traffic
IKEv2 has NAT-T built in natively; IKEv1 requires the RFC 3947 extension
Firewalls must allow UDP 500 and UDP 4500 (and optionally protocol 50 ESP if NAT-T is not needed) for IPsec to function

9. Cisco IOS IPsec Site-to-Site VPN Configuration

Complete configuration example for a site-to-site IPsec VPN using IKEv1 (ISAKMP) on Cisco IOS.

  Topology:
  HQ Router A                                    Branch Router B
  LAN: 192.168.1.0/24                            LAN: 192.168.2.0/24
  Public IP: 198.51.100.1                        Public IP: 203.0.113.1
  Outside IF: GigabitEthernet0/0                 Outside IF: GigabitEthernet0/0
  Inside IF:  GigabitEthernet0/1                 Inside IF:  GigabitEthernet0/1

Step 1 — Define Interesting Traffic (ACL)

  ! On BOTH routers (mirrored — what is src on one is dst on the other)

  ! Router A:
  access-list 100 permit ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255

  ! Router B:
  access-list 100 permit ip 192.168.2.0 0.0.0.255 192.168.1.0 0.0.0.255

Step 2 — Configure IKE Phase 1 Policy (ISAKMP)

  ! On BOTH routers (parameters must match exactly):
  crypto isakmp policy 10
   encr aes 256            ! Encryption: AES-256
   hash sha256             ! Integrity: SHA-256
   authentication pre-share ! Authentication: pre-shared key
   group 14               ! DH Group: 2048-bit (minimum recommended)
   lifetime 86400          ! SA lifetime: 24 hours (seconds)

Step 3 — Set the Pre-Shared Key

  ! Router A — key for peer Router B's public IP:
  crypto isakmp key Sup3rS3cr3tK3y address 203.0.113.1

  ! Router B — key for peer Router A's public IP:
  crypto isakmp key Sup3rS3cr3tK3y address 198.51.100.1

  ! Keys must be IDENTICAL on both sides

Step 4 — Define the IPsec Transform Set (Phase 2)

  ! On BOTH routers (must match):
  crypto ipsec transform-set MYSET esp-aes 256 esp-sha256-hmac
  !                            ↑ ESP     ↑ AES-256 encryption  ↑ SHA-256 integrity
   mode tunnel                 ! Tunnel mode (default for site-to-site)

Step 5 — Create the Crypto Map

  ! Router A:
  crypto map MYMAP 10 ipsec-isakmp
   set peer 203.0.113.1          ! Remote peer's public IP
   set transform-set MYSET       ! Reference the transform set from Step 4
   match address 100             ! Reference the ACL from Step 1
   set pfs group14               ! (Optional) Enable Perfect Forward Secrecy

  ! Router B:
  crypto map MYMAP 10 ipsec-isakmp
   set peer 198.51.100.1
   set transform-set MYSET
   match address 100
   set pfs group14

Step 6 — Apply Crypto Map to the Outside Interface

  ! On BOTH routers — apply to the WAN/outside interface:
  interface GigabitEthernet0/0
   crypto map MYMAP

Step 7 — Exempt VPN Traffic from NAT (if NAT is configured)

  ! Router A — VPN traffic must NOT be NATed:
  ! Deny VPN traffic in the NAT ACL (traffic that should NOT be NATed):
  ip access-list extended NAT_ACL
   deny   ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255
   permit ip 192.168.1.0 0.0.0.255 any

  ip nat inside source list NAT_ACL interface GigabitEthernet0/0 overload
  ! The "deny" line ensures 192.168.1.x → 192.168.2.x traffic bypasses NAT

10. Verification Commands

! ── Phase 1 (ISAKMP SA) Status ──────────────────────────────────────────
Router# show crypto isakmp sa

Expected output when tunnel is up:
IPv4 Crypto ISAKMP SA
dst src state conn-id status
203.0.113.1 198.51.100.1 QM_IDLE 1001 ACTIVE
! ↑ QM_IDLE = Phase 1 complete and ready

! ── Phase 2 (IPsec SA) Status ────────────────────────────────────────────
Router# show crypto ipsec sa

Key output to examine:
interface: GigabitEthernet0/0
Crypto map tag: MYMAP, local addr 198.51.100.1
protected vrf: (none)
local ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
current_peer 203.0.113.1 port 500
#pkts encaps: 143, #pkts encrypt: 143, #pkts digest: 143
#pkts decaps: 138, #pkts decrypt: 138, #pkts verify: 138
! ↑ Incrementing encaps/decaps = traffic flowing through tunnel

! ── Other Useful Commands ─────────────────────────────────────────────────
show crypto map ! Verify crypto map config
show crypto isakmp policy ! Show Phase 1 policies configured
show crypto ipsec transform-set ! Show transform sets configured
show crypto engine connections active ! Active encryption sessions
show crypto session ! IKEv2 session summary

! ── Debug Commands (use with caution in production) ──────────────────────
debug crypto isakmp ! Phase 1 negotiation in real time
debug crypto ipsec ! Phase 2 negotiation in real time
undebug all ! ALWAYS turn off debugging when done

11. IPsec Packet Flow — End to End

  PC at HQ (192.168.1.10) sends packet to PC at Branch (192.168.2.10):

  Step 1: PC sends normal IP packet to its default gateway (Router A)
          Src: 192.168.1.10, Dst: 192.168.2.10

  Step 2: Router A checks routing table → 192.168.2.0/24 reachable via VPN
          Router A checks crypto map ACL → matches access-list 100 → PROTECT

  Step 3: Router A looks up IPsec SA for this traffic (dst=203.0.113.1)
          If SA exists → proceed to Step 4
          If no SA → trigger IKE negotiation (Phase 1 + Phase 2) first

  Step 4: Router A applies ESP tunnel mode:
          - Encrypts original IP packet (AES-256)
          - Calculates HMAC-SHA256 over encrypted data
          - Adds ESP header (SPI + sequence number) and ESP trailer
          - Adds new outer IP header: src=198.51.100.1, dst=203.0.113.1
          - Increments sequence number (anti-replay)

  Step 5: Encrypted packet transmitted across internet
          Outer packet: src=198.51.100.1, dst=203.0.113.1, proto=ESP(50)

  Step 6: Router B receives packet, sees proto=50 (ESP)
          Looks up SPI from ESP header → finds matching SA in SAD
          Verifies HMAC integrity → if mismatch, discard
          Checks sequence number → if replay, discard
          Decrypts payload → recovers original IP packet

  Step 7: Router B forwards original packet to PC at Branch (192.168.2.10)
          PC receives packet as if sent directly from 192.168.1.10

12. Troubleshooting IPsec

Symptom	Likely Cause	Diagnostic & Fix
Phase 1 never establishes (`show crypto isakmp sa` shows MM_NO_STATE or empty)	Mismatched Phase 1 policy (encryption, hash, DH group, lifetime); UDP 500 blocked by firewall; routing problem to peer	Verify Phase 1 parameters match identically on both peers; confirm UDP 500 is not blocked; ping peer's public IP; `debug crypto isakmp` for detailed negotiation
Phase 1 up but Phase 2 never establishes (`show crypto ipsec sa` shows 0 packets)	Mismatched transform set; mismatched or non-mirrored ACL (interesting traffic); crypto map not applied to interface; PFS group mismatch	Verify transform sets match; verify ACLs are mirrored (HQ src/dst reversed on branch); confirm `crypto map` on correct interface; `debug crypto ipsec`
Tunnel establishes but traffic not flowing (encaps increment, decaps = 0)	Routing issue — traffic not being sent through tunnel; NAT not exempted; ACL on inside interface blocking	Check routing table — is 192.168.2.0/24 pointing toward correct interface? Verify NAT exemption ACL denies VPN traffic before NAT rule
Tunnel works from one side only	Asymmetric ACL — ACL on one side does not mirror the other; pre-shared key entered differently on each side	Verify ACL is mirrored; verify PSK is identical (no typos, case-sensitive)
IPsec behind NAT not working	NAT-T not supported or disabled; firewall blocking UDP 4500; AH being used instead of ESP	Ensure UDP 4500 is permitted; switch to ESP if AH configured; IKEv1 requires `crypto isakmp nat-traversal` command; IKEv2 handles automatically
Tunnel drops periodically and re-establishes	SA lifetime mismatch causing simultaneous re-keying conflict; DPD (Dead Peer Detection) timeout; unstable WAN link	Match SA lifetimes on both peers; ensure DPD timers are appropriate; check WAN stability

13. IPsec Use Cases

Use Case	Mode	IKE Version	Description
Site-to-Site VPN	Tunnel (gateway to gateway)	IKEv1 or IKEv2	Two routers/firewalls create a permanent encrypted tunnel; all traffic between the two LANs is protected automatically; users/PCs need no VPN client
Remote-Access VPN	Tunnel (client to gateway)	IKEv2 (preferred), IKEv1	Individual users run a VPN client (Cisco AnyConnect, native OS IKEv2); tunnel established from user's device to corporate gateway; IKEv2 supports EAP authentication (username/password + cert)
Host-to-Host	Transport (end to end)	IKEv2	Two servers directly establish IPsec; encrypts traffic between specific hosts; used for server-to-server replication, database encryption, or Windows IPsec policies
Cloud Connectivity	Tunnel	IKEv2	On-premises router connects to cloud VPN gateway (AWS VGW, Azure VPN Gateway, GCP Cloud VPN) — extends corporate network into IaaS environments

14. Key Points & Exam Tips

IPsec operates at Layer 3 — protects all IP traffic transparently to applications, unlike TLS which is application-layer.
AH (Protocol 51): Integrity + authentication, NO encryption. Incompatible with NAT. Rarely used. ESP (Protocol 50): Encryption + integrity + authentication. Widely used.
Transport mode: Protects payload only; original IP header visible. Used for host-to-host. Tunnel mode: Protects entire original packet; new outer IP header added. Used for site-to-site VPN (default).
IPsec SAs are unidirectional — two SAs needed for bidirectional communication. SPI identifies which SA to use for each packet.
IKE Phase 1 (ISAKMP SA): Builds secure channel to protect Phase 2. Parameters: encryption, hash, auth, DH group, lifetime. IKE Phase 2 (IPsec SA): Negotiates actual data protection. Parameters: ESP/AH, encryption, hash, mode, traffic selectors, PFS.
Phase 1 and Phase 2 parameters must match exactly on both peers — any mismatch causes negotiation failure.
IKEv2 is faster (4 messages vs 9), has built-in NAT-T, supports EAP, and is preferred for all new deployments.
DH Group 2 (1024-bit) is deprecated — use Group 14 (2048-bit) minimum; Groups 19/20 (ECDH) preferred.
PFS performs a new DH exchange for each Phase 2 SA — compromise of one session's keys does not reveal other sessions' keys.
NAT-T encapsulates ESP in UDP port 4500 — required when IPsec peers are behind NAT devices. IKE detection on UDP 500; data on UDP 4500.
Firewall must permit: UDP 500 (IKE), UDP 4500 (NAT-T), Protocol 50 (ESP) if NAT-T not used.
Verify with: show crypto isakmp sa (Phase 1), show crypto ipsec sa (Phase 2 + packet counts).
Always configure NAT exemption for VPN traffic when NAT is configured — VPN traffic must be excluded from the NAT ACL.

15. IPsec Fundamentals Quiz

1. A network engineer configures an IPsec site-to-site VPN. After deployment, traffic in one direction flows fine but traffic initiated from the branch cannot reach HQ. Phase 1 is established on both sides. What is the most likely cause?

A. The encryption algorithm does not match between peers B. NAT-T is not configured and the tunnel cannot form C. The DH group is deprecated and has been rejected D. The interesting traffic ACL is not correctly mirrored — the branch ACL must reverse the source and destination compared to the HQ ACL; a non-mirrored ACL causes traffic to only flow in the direction whose ACL matches

Correct answer is D. IPsec uses crypto ACLs (traffic selectors) to identify which packets should be encrypted. For a tunnel between HQ (192.168.1.0/24) and Branch (192.168.2.0/24), the ACLs must be mirrored: HQ ACL = "permit ip 192.168.1.0/24 192.168.2.0/24"; Branch ACL = "permit ip 192.168.2.0/24 192.168.1.0/24". If the branch ACL is incorrect (wrong source/destination, wrong subnet), traffic initiated from the branch does not match the ACL and is not encrypted — it goes out as plain IP and is either dropped or arrives at HQ unencrypted where the tunnel won't accept it. Phase 1 would still be established because Phase 1 only builds the management channel; Phase 2 may establish but with no matching selectors for branch-initiated traffic.

2. An IPsec VPN using AH (Authentication Header) is deployed between two sites. One site is behind a corporate firewall that performs NAT. The tunnel never establishes from that site. Why does AH fail with NAT, and what is the correct fix?

A. AH uses a different port than ESP and NAT blocks it by default B. AH calculates an integrity hash over the original IP source address — when NAT rewrites this address, the receiving peer's integrity check fails and packets are dropped. The fix is to switch from AH to ESP with NAT-T C. AH requires a public IP on both ends — NAT prevents this D. AH can be made to work with NAT by enabling NAT-T on UDP port 4500

Correct answer is B. This is the fundamental AH + NAT incompatibility. The Authentication Header calculates an HMAC integrity value that covers certain fields of the original IP header, including the source IP address. When a NAT device translates the source IP (e.g., from 192.168.1.1 to 1.2.3.4), the source IP field that AH protected changes. The receiving peer recalculates the HMAC over the received packet (with the NATted source IP) and compares it to the AH header — they don't match, so the packet is silently discarded. This cannot be fixed with NAT-T because NAT-T works by encapsulating ESP in UDP, which preserves the integrity of the inner ESP packet. AH has no equivalent mechanism. The only fix is to replace AH with ESP. ESP's integrity check covers only the ESP header and payload — not the outer IP header that NAT modifies.

3. What is the difference between IKE Phase 1 and Phase 2, and why are two separate phases needed?

A. Phase 1 encrypts data; Phase 2 authenticates the peers B. Phase 1 establishes the IPsec data SAs; Phase 2 negotiates IKE parameters C. Phase 1 creates a secure authenticated management channel (ISAKMP SA) used to safely negotiate Phase 2; Phase 2 negotiates the actual IPsec SAs that protect user data — separating management from data protection enables the fast Phase 2 Quick Mode to reuse the Phase 1 security context D. Phase 1 handles inbound traffic; Phase 2 handles outbound traffic

Correct answer is C. The two-phase design is fundamental to understanding IPsec. Phase 1 solves a chicken-and-egg problem: to negotiate IPsec parameters securely, the peers first need a secure channel. Phase 1 (ISAKMP SA / Main Mode — 6 messages in IKEv1) builds this secure, authenticated management channel using a Diffie-Hellman key exchange and peer authentication (PSK or certificates). Phase 2 (Quick Mode — 3 messages) then runs inside this already-protected Phase 1 channel to negotiate the actual IPsec SAs (encryption, integrity, traffic selectors, mode). The Phase 1 SA persists (default 24 hours) and can be reused for multiple Phase 2 negotiations — if a Phase 2 SA expires (default 1 hour), it can be renegotiated using the existing Phase 1 SA without full re-authentication. This is more efficient than repeating the full authentication exchange every hour.

4. An engineer is reviewing an IPsec configuration and sees that DH Group 2 is configured for both Phase 1 and Phase 2. A security audit flags this as a critical finding. What is the risk, and what should it be changed to?

A. DH Group 2 uses 1024-bit keys — considered cryptographically weak and potentially vulnerable to nation-state decryption. It should be replaced with DH Group 14 (2048-bit MODP) as a minimum, or Groups 19/20 (ECDH) for higher security and performance B. DH Group 2 is fine — it uses 1024-bit keys which are equivalent to AES-256 strength C. DH Group 2 is flagged because it is too secure and causes excessive CPU load — downgrade to Group 1 D. DH Group 2 should only be used in Phase 2 — in Phase 1 it is not supported

Correct answer is A. DH Group 2 uses 1024-bit MODP (Modular Exponential) keys. In 2015, researchers published the Logjam attack demonstrating that 1024-bit DH was within range of nation-state computational capabilities — the NSA was suspected of having pre-computed discrete logarithm tables for common 1024-bit primes. NIST deprecated 1024-bit DH in 2010 (SP 800-131A). For IPsec, the minimum recommended group is now Group 14 (2048-bit MODP). For best practice, use ECDH groups: Group 19 (P-256, 256-bit key equivalent to ~3072-bit RSA) or Group 20 (P-384) provide stronger security with smaller key sizes and faster computation than MODP groups. Groups 19 and 20 are also required for NSA Suite B / CNSA compliance.

5. A VPN engineer configures Perfect Forward Secrecy (PFS) on a high-throughput IPsec tunnel with a Phase 2 lifetime of 1 hour. The network team reports intermittent performance drops every hour. What is causing this, and how should PFS be evaluated?

A. PFS causes Phase 1 to re-establish every hour, requiring full re-authentication B. PFS is incompatible with the current DH group and should be disabled entirely C. PFS performs a new Diffie-Hellman computation during each Phase 2 re-key (every hour). This is CPU-intensive and causes brief performance impact. The trade-off: PFS provides stronger key isolation but at a computational cost; for very high-throughput tunnels, either extend Phase 2 lifetime to reduce re-key frequency, or use ECDH groups (19/20) which are much faster than MODP groups D. The drops are caused by Phase 2 SA expiry, not PFS — disabling PFS will fix the drops

Correct answer is C. Without PFS, Phase 2 re-keying reuses the existing Phase 1 Diffie-Hellman keying material — computationally cheap. With PFS enabled, each Phase 2 re-key performs a completely new DH key exchange, generating fresh cryptographic material from scratch. The DH computation (especially with larger MODP groups like Group 14 — 2048-bit) is CPU-intensive. On a high-throughput tunnel that re-keys every hour, this creates a noticeable performance spike during re-keying. The practical solutions: (1) Extend Phase 2 lifetime to 4–8 hours to reduce re-key frequency; (2) Switch to ECDH groups (19 or 20) which are orders of magnitude faster than same-security-level MODP groups; (3) Ensure the router has hardware crypto acceleration. Disabling PFS solves the performance issue but weakens key isolation.

6. A remote-access VPN client connects from a hotel network behind a NAT device. The VPN client uses IKEv1 with ESP. After some troubleshooting it is found that IKE Phase 1 completes but Phase 2 fails, and ESP packets are being dropped at the hotel firewall. What enables IPsec to work through this NAT environment?

A. Switching from ESP to AH — AH works better with NAT devices B. NAT Traversal (NAT-T) — detected during Phase 1 via NAT-D payloads; both peers switch to UDP 4500 encapsulation for all ESP traffic; the hotel firewall passes UDP 4500 where it was dropping raw ESP (protocol 50) C. Enabling PFS forces the tunnel to use a NAT-compatible cipher suite D. Changing to IKEv2 is the only solution — IKEv1 cannot work through NAT

Correct answer is B. This is the canonical NAT-T use case. The hotel's NAT device and firewall have two problems with raw ESP: (1) ESP (IP Protocol 50) has no port numbers — NAT devices using PAT cannot track or demultiplex it to the correct internal host. (2) Some firewalls block Protocol 50 outright. NAT-T solves both: during Phase 1, NAT-D (NAT Discovery) payloads detect the presence of NAT between the peers. Both peers then negotiate to encapsulate all ESP traffic inside UDP datagrams on port 4500. This gives the NAT device port numbers to track (UDP 4500 → specific client), and since UDP 4500 is standard traffic, most firewalls allow it. IKEv1 supports NAT-T via RFC 3947 extension (enabled with crypto isakmp nat-traversal on Cisco IOS); IKEv2 includes NAT-T natively.

7. An engineer runs `show crypto ipsec sa` and sees: `#pkts encaps: 500, #pkts encrypt: 500` on Router A, but on Router B: `#pkts decaps: 0, #pkts decrypt: 0`. What does this indicate?

A. The tunnel is working correctly — decaps on Router B count only inbound from the internet, not from Router A B. Router A is encrypting too many packets — reduce the interesting traffic ACL C. Phase 1 has not been established on Router B D. Router A is encrypting and sending packets but they are not reaching Router B — a transit network issue (routing, firewall blocking ESP/UDP 4500, MTU/fragmentation) is dropping the encrypted packets before they arrive

Correct answer is D. This output pattern is diagnostic: Router A's encaps/encrypt counters show traffic being encrypted and sent, but Router B's decaps/decrypt remain at zero — meaning the packets are never arriving at Router B. The IPsec SAs exist on both sides (otherwise Router A couldn't encrypt), but the transit network is dropping the packets. Common causes: (1) Firewall between the two routers blocking Protocol 50 (ESP) or UDP 4500 (NAT-T); (2) Routing asymmetry — encrypted packets taking a different path and getting dropped; (3) MTU issue — the added ESP+tunnel headers increase packet size; if PMTUD is broken, oversized packets are silently dropped. Diagnostic steps: traceroute from Router A's outside interface to Router B's outside IP; check intermediate firewall rules for ESP/UDP 4500; test with smaller pings to rule out MTU.

8. When configuring NAT exemption for a site-to-site VPN on a Cisco router, why must VPN traffic be excluded from the NAT ACL?

A. If VPN traffic is NATed, the source IP is translated before IPsec processes it — the crypto ACL match fails (no longer matches the interesting traffic definition) and the traffic is sent unencrypted over the internet instead of through the tunnel B. NAT breaks the IPsec tunnel by consuming additional bandwidth C. VPN traffic must be NATed before encryption — excluding it is incorrect D. NAT exemption is only required when using AH, not ESP

Correct answer is A. On a Cisco router, NAT is applied before IPsec encryption in the outbound packet flow. If the NAT ACL includes traffic destined for the VPN peer's subnet, the router NATs the source address (e.g., from 192.168.1.10 to 1.2.3.4) before checking whether it matches the crypto ACL. The crypto ACL says "protect traffic from 192.168.1.0/24" — but after NAT the source is now 1.2.3.4, which does not match. The traffic is sent unencrypted to the internet — a serious security failure. The fix is to explicitly deny VPN-bound traffic in the NAT ACL (route-map or extended ACL with a deny entry for the VPN subnets) before the permit any line. This ensures: VPN traffic → bypasses NAT → matches crypto ACL → encrypted. Internet traffic → matches NAT ACL → NATed → sent unencrypted to internet.

9. An organisation is migrating from IKEv1 to IKEv2 for all site-to-site VPNs. What are the primary technical advantages of IKEv2 that justify the migration effort?

A. IKEv2 supports stronger encryption than IKEv1 — IKEv1 cannot use AES-256 B. IKEv2 is faster because it eliminates all Diffie-Hellman operations C. IKEv2 uses 4 messages instead of 9 (faster establishment), has built-in NAT-T (no separate extension needed), supports EAP authentication (enabling username/password for remote access), includes MOBIKE (tunnel survives IP changes), and has built-in Dead Peer Detection — all previously requiring extensions or workarounds in IKEv1 D. IKEv2 eliminates the need for Phase 2 entirely — IPsec SAs are negotiated in Phase 1 only

Correct answer is C. IKEv2 (RFC 7296, 2014) was a comprehensive redesign of IKEv1. Key improvements: (1) Efficiency — IKEv1 requires 9 messages (6 for Main Mode + 3 for Quick Mode) to establish a tunnel; IKEv2 requires only 4 (IKE_SA_INIT + IKE_AUTH), halving connection establishment time. (2) NAT-T built-in — no need for the separate RFC 3947 extension or manual configuration; IKEv2 detects and handles NAT automatically. (3) EAP authentication — IKEv2 natively supports EAP, enabling remote access VPN with username/password authentication (without needing a separate XAUTH extension). (4) MOBIKE (RFC 4555) — allows established IKEv2 tunnels to survive IP address changes (e.g., mobile user switching from Wi-Fi to LTE). (5) Reliability — IKEv2 has built-in reliability with acknowledgements; IKEv1 relied on timers.

10. An IPsec VPN engineer needs to verify whether Phase 2 traffic is actually flowing through the tunnel after establishing it. Which command provides the most direct evidence, and what specific output confirms bidirectional traffic flow?

A. show crypto isakmp sa — state QM_IDLE confirms traffic is flowing B. show crypto ipsec sa — incrementing #pkts encaps and #pkts decaps counters on both peers confirm that traffic is being encrypted outbound and decrypted inbound, proving bidirectional data flow through the tunnel C. show crypto map — confirms the map is applied and traffic is flowing D. ping to the remote peer's public IP — a successful ping proves the tunnel carries data

Correct answer is B.

show crypto ipsec
                  sa

is the definitive command for confirming Phase 2 data flow. The critical counters: #pkts encaps / #pkts encrypt — packets that matched the crypto ACL and were encrypted outbound. These incrementing means the router is finding matching traffic and applying IPsec. #pkts decaps / #pkts decrypt — packets received from the peer and successfully decrypted. Both sets incrementing on both routers confirms true bidirectional traffic flow. In contrast: show crypto isakmp sa only confirms Phase 1 is established (QM_IDLE = ready for Phase 2, not that Phase 2 has traffic);

show
                  crypto map

confirms configuration; pinging the remote peer's public IP does not traverse the IPsec tunnel (VPN only encrypts traffic matching the interesting traffic ACL — the peer's public IP usually does not match).

← Back to Home

IPsec – ESP, AH, IKE Phases, Tunnel Mode & VPN Configuration

1. What Is IPsec?

2. IPsec Protocol Components: AH and ESP

Authentication Header (AH) — Protocol 51

Encapsulating Security Payload (ESP) — Protocol 50

AH vs ESP — Full Comparison

3. IPsec Modes: Transport vs Tunnel

Transport Mode

Tunnel Mode

4. Security Associations (SA) and the SAD

5. IKE — Internet Key Exchange

IKE Phase 1 — Building the Management Channel (ISAKMP SA)

IKE Phase 2 — Negotiating the IPsec SAs (Quick Mode)

6. IKEv1 vs IKEv2

7. Diffie-Hellman Key Exchange and Perfect Forward Secrecy

Diffie-Hellman (DH) Key Exchange

DH Group Recommendations

Perfect Forward Secrecy (PFS)

8. NAT Traversal (NAT-T)

9. Cisco IOS IPsec Site-to-Site VPN Configuration

Step 1 — Define Interesting Traffic (ACL)

Step 2 — Configure IKE Phase 1 Policy (ISAKMP)

Step 3 — Set the Pre-Shared Key

Step 4 — Define the IPsec Transform Set (Phase 2)

Step 5 — Create the Crypto Map

Step 6 — Apply Crypto Map to the Outside Interface

Step 7 — Exempt VPN Traffic from NAT (if NAT is configured)

10. Verification Commands

11. IPsec Packet Flow — End to End

12. Troubleshooting IPsec

13. IPsec Use Cases

14. Key Points & Exam Tips

15. IPsec Fundamentals Quiz

1. A network engineer configures an IPsec site-to-site VPN. After deployment, traffic in one direction flows fine but traffic initiated from the branch cannot reach HQ. Phase 1 is established on both sides. What is the most likely cause?

2. An IPsec VPN using AH (Authentication Header) is deployed between two sites. One site is behind a corporate firewall that performs NAT. The tunnel never establishes from that site. Why does AH fail with NAT, and what is the correct fix?

3. What is the difference between IKE Phase 1 and Phase 2, and why are two separate phases needed?

4. An engineer is reviewing an IPsec configuration and sees that DH Group 2 is configured for both Phase 1 and Phase 2. A security audit flags this as a critical finding. What is the risk, and what should it be changed to?

5. A VPN engineer configures Perfect Forward Secrecy (PFS) on a high-throughput IPsec tunnel with a Phase 2 lifetime of 1 hour. The network team reports intermittent performance drops every hour. What is causing this, and how should PFS be evaluated?

7. An engineer runs show crypto ipsec sa and sees: #pkts encaps: 500, #pkts encrypt: 500 on Router A, but on Router B: #pkts decaps: 0, #pkts decrypt: 0. What does this indicate?

8. When configuring NAT exemption for a site-to-site VPN on a Cisco router, why must VPN traffic be excluded from the NAT ACL?

9. An organisation is migrating from IKEv1 to IKEv2 for all site-to-site VPNs. What are the primary technical advantages of IKEv2 that justify the migration effort?

10. An IPsec VPN engineer needs to verify whether Phase 2 traffic is actually flowing through the tunnel after establishing it. Which command provides the most direct evidence, and what specific output confirms bidirectional traffic flow?

7. An engineer runs `show crypto ipsec sa` and sees: `#pkts encaps: 500, #pkts encrypt: 500` on Router A, but on Router B: `#pkts decaps: 0, #pkts decrypt: 0`. What does this indicate?