EEM — Embedded Event Manager Scripting
Every network engineer has experienced the same situation: an interface flaps at 2 AM, a CPU spike causes packet drops, or a BGP peer resets — and by the time anyone is alerted and investigates, the transient event is gone and the evidence has rolled out of the syslog buffer. Embedded Event Manager (EEM) is Cisco IOS's built-in automation engine that eliminates this problem. EEM allows you to define policy rules directly on the router — when a specified event occurs, IOS automatically executes a set of actions without any human intervention, without an external NMS, and without waiting for someone to notice an alert. For the broader automation landscape see Network Automation Overview.
EEM policies are called applets — small, self-contained event-action rules written directly in IOS global configuration. An applet has two parts: an event clause that defines what triggers it (a syslog message pattern, an interface state change, a CLI command being entered, a CPU threshold, an IP SLA state change, or a timer), and one or more action clauses that define what happens in response (send a syslog message, execute CLI commands, send an email, reload the device, or capture show command output to a file).
Before starting this lab, ensure you understand syslog message formats at Syslog Severity Levels and syslog server configuration at Syslog Configuration. For IP SLA probes that EEM can monitor as event triggers, see IP SLA Configuration & Tracking. For SSH access used by EEM to send alerts, see SSH Configuration.
1. EEM Architecture — Core Concepts
EEM Components
| Component | Description | IOS Keyword |
|---|---|---|
| Event Detector | Monitors the IOS subsystem for a specific condition — syslog message pattern, interface state, CPU level, timer expiry, CLI command, IP SLA state, SNMP OID value | event [detector-type] |
| Policy (Applet) | The complete event-action rule. Named by the administrator. Stored in running-config and survives reload when saved. An applet can have one event and multiple ordered actions | event manager applet [name] |
| Action | What the router does when the event fires — send syslog, run CLI commands, send email, set a variable, evaluate an expression, or call another applet | action [label] [action-type] |
| Environment Variable | A global key-value variable available to all EEM applets — used to store email recipients, threshold values, or frequently changed parameters without editing the applet itself | event manager environment [name] [value] |
| Action Label | A number (or string) that defines the execution order of actions within an applet. Actions execute in ascending label order. Labels are typically 1.0, 2.0, 3.0 — decimals allow inserting actions between existing ones | action 1.0 ... |
EEM Event Detectors
| Event Detector | Trigger Condition | Common Use Case |
|---|---|---|
event syslog |
A syslog message matching a specified pattern (regex) is generated by IOS | Interface down/up, BGP peer reset, authentication failure, any IOS-generated log message |
event interface |
An interface counter crosses a threshold (input errors, output drops, bandwidth utilisation) | Detect high error rates, CRC errors, or queue drops on a WAN interface |
event track |
An IP SLA tracking object changes state (Up to Down, or Down to Up) | Take action when a tracked WAN link or default route becomes unavailable |
event cli |
A specific CLI command is entered by a user in exec or config mode | Audit sensitive commands (no shut, write erase), alert on unauthorised changes, or block specific commands |
event timer |
A timer expires — periodic (every N seconds), absolute (specific date/time), or watchdog (fires unless reset) | Periodic configuration backup, scheduled reporting, periodic health-check commands |
event snmp |
An SNMP OID value crosses a threshold | CPU utilisation above 80%, memory free below threshold, interface errors exceed rate |
event ipsla |
An IP SLA operation returns a specific result (timeout, RTT threshold exceeded, jitter threshold exceeded) | Detect WAN latency degradation, trigger path switchover when probe fails |
event none |
No automatic trigger — applet runs only when manually triggered with event manager run [name] |
On-demand maintenance scripts, test applets, manually invoked diagnostics |
EEM Action Types
| Action Type | What It Does | Example |
|---|---|---|
action syslog |
Generates a custom syslog message at a specified severity level | Log "Interface down alert" at severity 2 (Critical) |
action cli command |
Executes a CLI command in exec or configuration mode as if typed by an operator | Run show interfaces, clear ip bgp, or any config command |
action mail |
Sends an email via a configured SMTP server with a custom subject and body | Email NOC team when an interface goes down |
action puts |
Writes a string to the console, a file, or a buffer — useful for logging applet output | Write "EEM triggered at [time]" to flash:/eem-log.txt |
action set |
Sets a local EEM variable to a value or expression result | Store the current interface status in a variable for conditional logic |
action info type |
Retrieves event-specific information — interface name, syslog message text, track state — and stores it in a variable | Capture which interface triggered the event, then use it in subsequent actions |
action reload |
Reloads the device — use with extreme caution | Automated recovery from a known-bad config state (rarely appropriate) |
EEM Applet Execution Flow
IOS generates event (e.g., interface goes down)
│
▼
EEM Event Detector scans all registered applets
for a matching event clause
│
▼ (match found)
EEM Scheduler queues the applet for execution
│
▼
Actions execute in ascending label order:
action 1.0 → action 2.0 → action 3.0 → ...
│
▼
Applet completes — EEM returns to monitoring state
IMPORTANT NOTES:
─────────────────────────────────────────────────────────
● By default, applets run synchronously in the IOS main
scheduler. Use "event [type] sync no" for async execution
on heavy applets to avoid blocking IOS processes.
● "action cli command" runs in a separate Tcl shell context.
It requires "event manager session cli username [user]"
or a configured privilege level to execute privileged commands.
● Actions that modify configuration take effect immediately —
there is no "review before apply" step. Test applets with
"event none" before attaching a live event trigger.
2. Lab Topology & Scenario
NetsTuts_R1 is an edge router with two WAN interfaces (Gi0/0 to ISP-A, Gi0/1 to ISP-B) and one LAN interface (Gi0/2). The NOC team needs automated alerts and diagnostic capture when WAN interfaces fail. Four EEM applets will be built in this lab:
ISP-A ISP-B
203.0.113.1 198.51.100.1
| |
Gi0/0 Gi0/1
\ /
\ /
[ NetsTuts_R1 ]
Gi0/2 (LAN)
10.0.0.1/24
EEM Applets to build:
┌───┬───────────────────────────────┬────────────────────────────────────┐
│ # │ Applet Name │ Purpose │
├───┼───────────────────────────────┼────────────────────────────────────┤
│ 1 │ INTF-DOWN-ALERT │ Syslog + show commands when │
│ │ │ any interface goes down │
├───┼───────────────────────────────┼────────────────────────────────────┤
│ 2 │ INTF-DOWN-EMAIL │ Email NOC team when WAN interface │
│ │ │ Gi0/0 goes down │
├───┼───────────────────────────────┼────────────────────────────────────┤
│ 3 │ CPU-THRESHOLD-ALERT │ Log and capture diagnostics when │
│ │ │ CPU exceeds 80% for 5 seconds │
├───┼───────────────────────────────┼────────────────────────────────────┤
│ 4 │ CONFIG-CHANGE-AUDIT │ Log who entered config mode and │
│ │ │ capture a config diff │
└───┴───────────────────────────────┴────────────────────────────────────┘
3. Step 1 — Prerequisites and EEM Global Settings
Before writing applets, configure the global EEM prerequisites: an EEM user for CLI action execution, the SMTP server for email actions, and environment variables for reusable parameters.
NetsTuts_R1>en NetsTuts_R1#conf t Enter configuration commands, one per line. End with CNTL/Z. ! ══════════════════════════════════════════════════════════ ! EEM CLI Execution User ! Required so "action cli command" can run privileged commands ! ══════════════════════════════════════════════════════════ ! ── Create a local user for EEM to impersonate ──────────── NetsTuts_R1(config)#username eem-user privilege 15 secret EEM$ecret99 ! ── Tell EEM to use this user for cli action sessions ───── NetsTuts_R1(config)#event manager session cli username eem-user ! ══════════════════════════════════════════════════════════ ! SMTP Mail Server (for action mail) ! ══════════════════════════════════════════════════════════ NetsTuts_R1(config)#event manager environment _email_server 10.0.0.25 NetsTuts_R1(config)#event manager environment _email_from [email protected] NetsTuts_R1(config)#event manager environment _email_to [email protected] NetsTuts_R1(config)#event manager environment _email_cc [email protected] ! ══════════════════════════════════════════════════════════ ! EEM Environment Variables — reusable parameters ! Change these without editing each applet individually ! ══════════════════════════════════════════════════════════ NetsTuts_R1(config)#event manager environment _cpu_threshold 80 NetsTuts_R1(config)#event manager environment _hostname NetsTuts_R1 NetsTuts_R1(config)#event manager environment _log_dir flash:/eem-logs/
action cli command. Without it, the
CLI action runs with minimal privilege and cannot execute
show commands, enter configuration mode, or clear
processes. The environment variables prefixed with
an underscore (a Cisco convention for EEM variables) centralise
values that appear in multiple applets — changing the NOC email
address only requires updating the environment variable, not
editing each applet. For user account configuration, see
Hostname,
Banner & Password Configuration.
4. Step 2 — Applet 1: Interface Down Syslog Alert
This applet fires whenever any interface transitions to a down state.
The event trigger is a syslog detector matching the
standard IOS message generated for an interface going down.
Actions send a custom syslog alert and execute
show interfaces to capture a snapshot at the moment
of failure.
NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT NetsTuts_R1(config-applet)# description "Alert and capture diag when any intf goes down" ! ── EVENT: match IOS syslog message for interface down ──── ! ── %LINK-3-UPDOWN: Interface [intf], changed state to down NetsTuts_R1(config-applet)# event syslog pattern "%LINK-3-UPDOWN.*down" NetsTuts_R1(config-applet)# maxrun 60 ! ── ACTION 1: Send custom syslog at Critical severity ───── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical msg \ "EEM ALERT: Interface down event detected on $_hostname" ! ── ACTION 2: Get the syslog message that triggered the event NetsTuts_R1(config-applet)# action 2.0 info type routername NetsTuts_R1(config-applet)# action 2.1 set _rname "$_info_routername" ! ── ACTION 3: Run show interfaces brief — capture state ─── NetsTuts_R1(config-applet)# action 3.0 cli command "enable" NetsTuts_R1(config-applet)# action 3.1 cli command \ "show ip interface brief | redirect flash:/eem-logs/intf-brief.txt" ! ── ACTION 4: Run show interfaces for the triggering event─ NetsTuts_R1(config-applet)# action 4.0 cli command \ "show interfaces | redirect flash:/eem-logs/intf-detail.txt" ! ── ACTION 5: Log completion message ────────────────────── NetsTuts_R1(config-applet)# action 5.0 syslog priority informational \ msg "EEM INTF-DOWN-ALERT: diagnostic files saved to flash:/eem-logs/" NetsTuts_R1(config-applet)#exit
event syslog pattern uses a regular expression
matching the standard IOS link-state change message. The pattern
%LINK-3-UPDOWN.*down matches the facility
(LINK), severity (3), mnemonic
(UPDOWN), and the word down at the end —
it fires on any interface going down but not on interfaces coming
back up (which would say up). The maxrun 60
sets the maximum applet runtime to 60 seconds — if actions take
longer than this, EEM terminates the applet. The | redirect
pipe writes CLI output to a file — useful for post-mortem analysis
of what the router's state was at the exact moment of the fault.
See show interfaces for the
output captured by action 4.0.
For syslog message formats and severity levels, see
Syslog Severity Levels.
Understanding the Syslog Pattern Match
IOS syslog message format:
%FACILITY-SEVERITY-MNEMONIC: message text
Example messages that trigger INTF-DOWN-ALERT:
%LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
%LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
Pattern: "%LINK-3-UPDOWN.*down"
%LINK-3-UPDOWN ← literal match of facility-severity-mnemonic
.* ← any characters (interface name varies)
down ← ends with "down" — not "up"
Messages that do NOT trigger (interface coming back up):
%LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
^^^ "up" not "down"
More specific pattern (only Gi0/0):
"%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down"
Related messages to consider:
%LINEPROTO-5-UPDOWN: Line protocol on Interface ... changed state to down
↑ Layer 3 protocol down (after Layer 1 is already down)
Use severity 5 and LINEPROTO mnemonic for Layer 3 events
5. Step 3 — Applet 2: Interface Down Email Notification
This applet sends an email to the NOC team when the primary WAN interface (Gi0/0) specifically goes down. It uses EEM environment variables for the mail parameters — no email address is hard-coded in the applet itself.
NetsTuts_R1(config)#event manager applet INTF-DOWN-EMAIL NetsTuts_R1(config-applet)# description "Email NOC when primary WAN Gi0/0 goes down" ! ── EVENT: match only Gi0/0 going down ─────────────────── NetsTuts_R1(config-applet)# event syslog pattern \ "%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down" NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Syslog alert ──────────────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM EMAIL: Primary WAN Gi0/0 DOWN — alerting NOC via email" ! ── ACTION 2: Capture current time for email body ───────── NetsTuts_R1(config-applet)# action 2.0 info type snmp getid oid sysUpTime.0 NetsTuts_R1(config-applet)# action 2.1 set _uptime "$_info_snmp_value" ! ── ACTION 3: Send email using environment variables ────── NetsTuts_R1(config-applet)# action 3.0 mail server "$_email_server" \ to "$_email_to" \ from "$_email_from" \ cc "$_email_cc" \ subject "ALERT: Primary WAN Gi0/0 DOWN on $_hostname" \ body "Router $_hostname: Primary WAN interface GigabitEthernet0/0 \ has gone DOWN. Immediate investigation required. \ Router uptime: $_uptime. \ Check syslog server for full event details." ! ── ACTION 4: Log that email was sent ───────────────────── NetsTuts_R1(config-applet)# action 4.0 syslog priority informational \ msg "EEM EMAIL: Alert email sent to $_email_to for Gi0/0 down event" NetsTuts_R1(config-applet)#exit
action mail command requires the IOS device to have
IP connectivity to the SMTP server (_email_server).
The SMTP server must accept unauthenticated relay from the router's
source IP, or SMTP authentication credentials must be configured.
In most enterprise environments, a local relay (Postfix, Exchange
relay connector, or a syslog server with email forwarding) is easier
to configure than sending directly to an external SMTP server. Note
that the email server variable ($_email_server) and
recipient variables all reference the global environment variables
set in Step 1 — changing the NOC email address means updating only
the environment variable, not editing this applet. For syslog
server setup, see Syslog
Server Configuration.
6. Step 4 — Applet 3: CPU Threshold Alert
This applet uses the snmp event detector to monitor
the router's CPU utilisation OID. When the 5-second CPU average
exceeds 80%, it logs an alert and captures show processes cpu
sorted to identify the offending process.
NetsTuts_R1(config)#event manager applet CPU-THRESHOLD-ALERT NetsTuts_R1(config-applet)# description "Alert when 5s CPU exceeds 80 percent" ! ── EVENT: SNMP OID for 5-second CPU average ───────────── ! ── OID 1.3.6.1.4.1.9.2.1.56 = Cisco 5-second CPU % NetsTuts_R1(config-applet)# event snmp oid 1.3.6.1.4.1.9.2.1.56 \ get-type exact \ entry-op gt entry-val 80 \ poll-interval 10 NetsTuts_R1(config-applet)# maxrun 60 ! ── ACTION 1: Log critical syslog ───────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM CPU-ALERT: CPU utilisation exceeded 80% on $_hostname" ! ── ACTION 2: Capture CPU process list ─────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show processes cpu sorted | head 30 | \ redirect flash:/eem-logs/cpu-high.txt" ! ── ACTION 3: Capture memory summary ───────────────────── NetsTuts_R1(config-applet)# action 3.0 cli command \ "show processes memory sorted | head 20 | \ redirect flash:/eem-logs/mem-high.txt" ! ── ACTION 4: Log file location ─────────────────────────── NetsTuts_R1(config-applet)# action 4.0 syslog priority warning \ msg "EEM CPU-ALERT: Diagnostics saved to flash:/eem-logs/cpu-high.txt" NetsTuts_R1(config-applet)#exit
event snmp detector polls the SNMP OID every
poll-interval 10 seconds. entry-op gt
means "fire when the value is greater than" the
entry-val 80 threshold. SNMP must be enabled on the
router for the SNMP event detector to function — it uses the local
SNMP agent to read the OID, so a remote NMS is not required.
An alternative approach using the syslog detector:
IOS generates %SYS-4-CPUHOG messages when a process
runs for too long — matching this pattern with
event syslog pattern "%SYS-4-CPUHOG" works on devices
without SNMP configured. For SNMP configuration, see
SNMP v2c & v3
Configuration.
Alternative: CPU Alert Using Syslog Pattern (no SNMP required)
! ── Alternative CPU applet using syslog event detector ─── NetsTuts_R1(config)#event manager applet CPU-HOG-SYSLOG NetsTuts_R1(config-applet)# description "Alert on CPU hog process via syslog" NetsTuts_R1(config-applet)# event syslog pattern "%SYS-3-CPUHOG" NetsTuts_R1(config-applet)# maxrun 30 NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM: CPU HOG process detected on $_hostname — check processes" NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show processes cpu sorted | redirect flash:/eem-logs/cpuhog.txt" NetsTuts_R1(config-applet)#exit
7. Step 5 — Applet 4: Configuration Change Audit
This applet uses the cli event detector to fire
whenever a user enters global configuration mode. It logs the
username and terminal line, then captures a configuration diff
showing exactly what changed. This is a lightweight change-audit
trail without requiring an external change management system.
NetsTuts_R1(config)#event manager applet CONFIG-CHANGE-AUDIT NetsTuts_R1(config-applet)# description "Log who entered config mode and capture diff" ! ── EVENT: fire when 'configure terminal' is entered ───── NetsTuts_R1(config-applet)# event cli pattern "conf.*t" \ sync no \ skip no NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Retrieve who entered the command ──────────── NetsTuts_R1(config-applet)# action 1.0 info type cli username NetsTuts_R1(config-applet)# action 1.1 set _user "$_info_cli_username" ! ── ACTION 2: Get the terminal line information ─────────── NetsTuts_R1(config-applet)# action 2.0 info type tty_name NetsTuts_R1(config-applet)# action 2.1 set _tty "$_info_tty_name" ! ── ACTION 3: Log the configuration entry event ────────── NetsTuts_R1(config-applet)# action 3.0 syslog priority warning \ msg "EEM AUDIT: User '$_user' entered config mode via $_tty on $_hostname" ! ── ACTION 4: Capture running config snapshot before changes NetsTuts_R1(config-applet)# action 4.0 cli command "enable" NetsTuts_R1(config-applet)# action 4.1 cli command \ "show running-config | redirect flash:/eem-logs/config-before.txt" ! ── ACTION 5: Log the file location ────────────────────── NetsTuts_R1(config-applet)# action 5.0 syslog priority informational \ msg "EEM AUDIT: Pre-change config snapshot saved to flash" NetsTuts_R1(config-applet)#exit
event cli pattern "conf.*t" matches any CLI
command beginning with conf and containing
t — which covers configure terminal and
common abbreviations (conf t, config t).
sync no means the applet runs asynchronously — the
user's configure terminal command proceeds immediately
without waiting for the applet to finish (important so the audit
does not delay the operator). skip no means the
original command still executes — the applet observes it without
blocking it. Use skip yes to prevent the
command from executing (useful to block specific dangerous commands).
For running-config management, see
show running-config
and Saving
& Managing Cisco Configurations.
8. Additional Practical Applet Examples
Applet 5: Periodic Configuration Backup (Timer Event)
! ── Back up running-config to TFTP every 24 hours ───────── NetsTuts_R1(config)#event manager applet DAILY-CONFIG-BACKUP NetsTuts_R1(config-applet)# description "Copy running-config to TFTP daily" ! ── EVENT: periodic timer — fires every 86400 seconds (24h) NetsTuts_R1(config-applet)# event timer periodic interval 86400 NetsTuts_R1(config-applet)# maxrun 60 ! ── ACTION 1: Log the backup start ─────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority informational \ msg "EEM BACKUP: Starting daily config backup on $_hostname" ! ── ACTION 2: Copy to TFTP server ───────────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "copy running-config tftp://10.0.0.100/backups/NetsTuts_R1-backup.cfg" NetsTuts_R1(config-applet)# action 2.2 cli command "" ! ── Empty string confirms any TFTP dialog prompts ───────── ! ── ACTION 3: Log completion ────────────────────────────── NetsTuts_R1(config-applet)# action 3.0 syslog priority informational \ msg "EEM BACKUP: Daily config backup completed for $_hostname" NetsTuts_R1(config-applet)#exit
event timer periodic interval 86400 fires every
86,400 seconds (24 hours) from when the applet was last triggered.
On the first trigger after a reload, the timer starts from the
moment the applet was registered. The empty string action 2.2
cli command "" sends a carriage return to confirm any
interactive prompts the copy command might display
(such as destination filename confirmation). For configuration
backup and management, see
Saving &
Managing Cisco Configurations.
Applet 6: BGP Peer Down Alert
! ── Alert when any BGP peer goes down ───────────────────── NetsTuts_R1(config)#event manager applet BGP-PEER-DOWN NetsTuts_R1(config-applet)# description "Alert when BGP neighbour resets" ! ── EVENT: match BGP neighbour state change syslog ──────── ! ── %BGP-5-ADJCHANGE: neighbor x.x.x.x Down NetsTuts_R1(config-applet)# event syslog pattern \ "%BGP-5-ADJCHANGE.*Down" NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Critical syslog ───────────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM BGP-ALERT: BGP neighbour down on $_hostname — check immediately" ! ── ACTION 2: Capture BGP summary ──────────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show ip bgp summary | redirect flash:/eem-logs/bgp-down.txt" NetsTuts_R1(config-applet)# action 2.2 cli command \ "show ip bgp neighbors | redirect flash:/eem-logs/bgp-neighbors.txt" ! ── ACTION 3: Email NOC ─────────────────────────────────── NetsTuts_R1(config-applet)# action 3.0 mail server "$_email_server" \ to "$_email_to" \ from "$_email_from" \ subject "ALERT: BGP Peer DOWN on $_hostname" \ body "A BGP neighbour adjacency has gone down on $_hostname. \ BGP diagnostics saved to flash:/eem-logs/. Check BGP summary immediately." NetsTuts_R1(config-applet)#exit
Applet 7: IP SLA Tracking Object State Change
! ── React when IP SLA tracking object changes state ─────── ! ── Assumes: ip sla 1 configured to probe WAN gateway ! ── track 1 ip sla 1 reachability NetsTuts_R1(config)#event manager applet WAN-SLA-DOWN NetsTuts_R1(config-applet)# description "Alert when WAN SLA track object goes down" ! ── EVENT: track object 1 transitions to "down" state ──── NetsTuts_R1(config-applet)# event track 1 state down NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Log the WAN failure ───────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM WAN-SLA: Track object 1 DOWN — WAN probe failing on $_hostname" ! ── ACTION 2: Show IP SLA statistics ───────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show ip sla statistics 1 | redirect flash:/eem-logs/sla-down.txt" ! ── ACTION 3: Show routing table — verify failover ──────── NetsTuts_R1(config-applet)# action 3.0 cli command \ "show ip route | redirect flash:/eem-logs/route-at-failure.txt" NetsTuts_R1(config-applet)#exit
event track [number] state down detector integrates
directly with IOS Object Tracking — the same tracking objects used
by IP SLA, static route tracking, and HSRP. When the tracked object
transitions to down (probe fails), the applet fires immediately —
no syslog pattern matching delay. This is more precise than a
syslog pattern match for IP SLA events. A companion applet with
event track 1 state up can send an "all clear" alert
when the WAN recovers. For IP SLA and tracking configuration, see
IP SLA Configuration
& Tracking. For the combination of IP SLA + EEM for
automated WAN monitoring, see
IP SLA with Syslog Alerting.
9. Verification
show event manager policy registered — List All Applets
NetsTuts_R1#show event manager policy registered
No. Class Type Event Type Trap Time Registered
--- -------- ------ ------------------ ---- ----------------
1 applet system syslog Off Wed Oct 16 09:00:01 2024
Name : INTF-DOWN-ALERT
Event : pattern {%LINK-3-UPDOWN.*down}
Description : Alert and capture diag when any intf goes down
2 applet system syslog Off Wed Oct 16 09:00:02 2024
Name : INTF-DOWN-EMAIL
Event : pattern {%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down}
Description : Email NOC when primary WAN Gi0/0 goes down
3 applet system snmp Off Wed Oct 16 09:00:03 2024
Name : CPU-THRESHOLD-ALERT
Event : OID {1.3.6.1.4.1.9.2.1.56} op gt val {80}
Description : Alert when 5s CPU exceeds 80 percent
4 applet system cli Off Wed Oct 16 09:00:04 2024
Name : CONFIG-CHANGE-AUDIT
Event : pattern {conf.*t} sync {no} skip {no}
Description : Log who entered config mode and capture diff
5 applet system timer Off Wed Oct 16 09:00:05 2024
Name : DAILY-CONFIG-BACKUP
Event : periodic interval {86400}
Description : Copy running-config to TFTP daily
show event manager policy active — Applets Currently Running
NetsTuts_R1#show event manager policy active No. Applet name Class Event Type State --- ------------------- -------- ------------ ------- (none currently running) ! ── If an applet is mid-execution, it appears here ──────── ! ── with execution time and current action label ──────────
show event manager history events — Execution History
NetsTuts_R1#show event manager history events No. Time Event Type Name --- ------------------- ------------------ ------------------- 1 Wed Oct 16 14:32:01 syslog INTF-DOWN-ALERT 2 Wed Oct 16 14:32:01 syslog INTF-DOWN-EMAIL 3 Wed Oct 16 14:32:05 syslog INTF-DOWN-ALERT 4 Wed Oct 16 09:00:00 timer DAILY-CONFIG-BACKUP
show event manager history traps — SNMP Trap History
NetsTuts_R1#show event manager history traps No. Time Name Status --- ------------------- -------------------- ------- 1 Wed Oct 16 14:32:01 INTF-DOWN-ALERT success 2 Wed Oct 16 14:32:01 INTF-DOWN-EMAIL success
show event manager environment — Verify Environment Variables
NetsTuts_R1#show event manager environment Name Value -------------------------- --------------------------------- _email_server 10.0.0.25 _email_from [email protected] _email_to [email protected] _email_cc [email protected] _cpu_threshold 80 _hostname NetsTuts_R1 _log_dir flash:/eem-logs/
Manually Trigger an Applet — Testing with event none
! ── Step 1: temporarily change the event to "none" for testing NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT NetsTuts_R1(config-applet)# event none NetsTuts_R1(config-applet)#exit ! ── Step 2: manually run the applet ─────────────────────── NetsTuts_R1#event manager run INTF-DOWN-ALERT % EEM policy successfully executed ! ── Step 3: verify files were created ──────────────────── NetsTuts_R1#dir flash:/eem-logs/ Directory of flash:/eem-logs/ 1 -rw- 3512 Oct 16 2024 14:35:22 intf-brief.txt 2 -rw- 45672 Oct 16 2024 14:35:22 intf-detail.txt ! ── Step 4: restore the original syslog event trigger ───── NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT NetsTuts_R1(config-applet)# event syslog pattern "%LINK-3-UPDOWN.*down" NetsTuts_R1(config-applet)#exit ! ── Step 5: verify logs show the manual execution ───────── NetsTuts_R1#show event manager history events No. Time Event Type Name --- ------------------- ------------------ ------------------- 1 Wed Oct 16 14:35:22 none INTF-DOWN-ALERT
event none before
attaching a live trigger — this prevents the applet from firing
on real events during development and allows you to execute
event manager run at a controlled time. Verify all
actions complete correctly, check that files are created in the
expected locations, and confirm syslog messages appear with
show logging before switching to the production
event trigger.
show logging — Confirm EEM Syslog Actions
NetsTuts_R1#show logging | include EEM Oct 16 14:32:01: %HA_EM-2-LOG: INTF-DOWN-ALERT: EEM ALERT: Interface down event detected on NetsTuts_R1 Oct 16 14:32:01: %HA_EM-6-LOG: INTF-DOWN-ALERT: EEM INTF-DOWN-ALERT: diagnostic files saved to flash:/eem-logs/ Oct 16 14:32:01: %HA_EM-2-LOG: INTF-DOWN-EMAIL: EEM EMAIL: Primary WAN Gi0/0 DOWN — alerting NOC via email Oct 16 14:32:02: %HA_EM-6-LOG: INTF-DOWN-EMAIL: EEM EMAIL: Alert email sent to [email protected]
%HA_EM (High Availability Embedded Manager) and the
severity level specified in the action syslog priority
command. The applet name is included in the log message — making
it easy to filter and identify EEM-generated messages with
show logging | include EEM or
show logging | include HA_EM. For log buffer
management and forwarding EEM syslog to a central server, see
show logging and
Syslog Server
Configuration.
Verification Command Summary
| Command | What It Shows | Primary Use |
|---|---|---|
show event manager policy registered |
All registered applets — name, event type, pattern/threshold, registration time | Primary verification — confirm applet is registered with correct event trigger |
show event manager policy active |
Applets currently executing — name, state, runtime, current action label | Debug a running applet — see if it is stuck on a slow action (e.g., copy to TFTP timing out) |
show event manager history events |
Historical applet executions — which applet, when, and what event type triggered it | Confirm an applet fired when expected, or diagnose why it did not fire |
show event manager history traps |
SNMP trap history from EEM — applet name and success/failure status | Confirm SNMP-triggered applets executed and check for execution errors |
show event manager environment |
All configured EEM environment variables and their current values | Verify environment variables are set correctly before testing email or file path actions |
event manager run [name] |
Manually triggers an applet with event none configured |
Test applet actions safely without waiting for a real event to occur |
show logging | include HA_EM |
All EEM-generated syslog messages in the log buffer | Confirm action syslog commands are executing and producing the expected log messages |
dir flash:/eem-logs/ |
Files written to flash by EEM action cli command with | redirect |
Confirm diagnostic capture files are being created with correct timestamps |
10. Troubleshooting EEM Applets
| Problem | Symptom | Cause | Fix |
|---|---|---|---|
| Applet never triggers despite the event occurring | show event manager history events shows no executions for the applet — the interface went down but INTF-DOWN-ALERT never fired |
The syslog pattern in the event syslog pattern clause does not match the actual syslog message IOS generates. IOS syslog message text varies slightly between IOS versions and platforms |
Capture the exact syslog message from show logging at the moment the event occurs. Compare it character-by-character with the pattern in the applet. The pattern is a regular expression — test it with event none and manually verify the regex matches the log line. Remember that the % character in the pattern matches the literal % in the syslog message. Use debug event manager to see real-time EEM pattern matching activity |
action cli command fails — access denied or incomplete output |
Applet fires but the CLI commands produce no output files, or show event manager history shows the applet executed with errors. dir flash:/eem-logs/ shows no files created |
The EEM session CLI user is not configured (event manager session cli username is missing), or the configured user does not have privilege level 15, or the flash directory does not exist |
Verify event manager session cli username eem-user is in the running config. Confirm the user has privilege 15 with show running-config | include username eem-user. Create the log directory manually first: mkdir flash:/eem-logs from the router CLI. Re-test with event manager run [name] |
| Email action never sends emails | Applet fires and logs show action 3.0 mail executing, but no email arrives at the NOC inbox |
The SMTP server (_email_server) is not reachable from the router, the SMTP server does not accept relay from the router's source IP, or the environment variable contains an incorrect IP address |
Verify SMTP server reachability: ping 10.0.0.25 from the router. Confirm the SMTP server allows relay from the router's IP. Check show event manager environment to verify _email_server is correct. Try a simple Telnet test: telnet 10.0.0.25 25 — if port 25 does not respond, SMTP is blocked. In many environments it is easier to use a syslog-to-email gateway instead of direct SMTP |
| Applet triggers too frequently — log flooding | The applet fires dozens of times per minute — syslog is flooded with EEM messages and flash storage fills up quickly with captured files | The triggering event is recurring very rapidly (e.g., an interface flapping up and down repeatedly) and the applet has no rate limiting or cooldown period | Add a rate limit to the event detector: event syslog pattern "..." maxrun 60 ratelimit 300 — the ratelimit 300 prevents the applet from firing again for 300 seconds (5 minutes) after its last execution. Alternatively use a event timer watchdog pattern to only fire once until manually reset. Investigate the root cause of the flapping interface separately |
| Applet actions execute out of order or skip actions | The applet fires but only some actions execute — for example, action 1.0 syslog works but action 2.0 cli command and 3.0 cli command do not run | The maxrun timer expired before all actions completed — IOS terminates the applet at the maxrun limit. CLI commands that capture large outputs (e.g., show processes cpu sorted | redirect) can take 5–15 seconds each |
Increase the maxrun value: for applets with multiple CLI capture commands, set maxrun 120 or higher. Verify the actual runtime with show event manager policy active during execution or review show event manager history events for timeout indicators. Remove non-critical actions to reduce runtime |
Config-audit applet blocks the operator's configure terminal |
When an engineer types conf t, the prompt hangs for 20–30 seconds before entering config mode — the EEM applet is running synchronously and blocking IOS |
The CLI event applet was configured with sync yes (or the default sync behaviour) — the applet runs synchronously, meaning IOS waits for the applet to complete before processing the matched command |
Change to asynchronous execution: event cli pattern "conf.*t" sync no skip no. With sync no, the operator's command executes immediately and the applet runs in the background. This is the correct setting for audit applets that should observe without blocking. Reserve sync yes only for applets that must complete before the matched command is processed (e.g., an applet that validates or modifies the command) |
Key Points & Exam Tips
- EEM (Embedded Event Manager) is an IOS-native automation engine that allows policy rules called applets to be defined directly on the router. Each applet has one
eventclause (trigger) and one or moreactionclauses (responses), executed in ascending label order. - The most common event detector is
event syslog pattern, which uses a regular expression to match IOS-generated syslog messages. The pattern must match the exact format of the syslog message including facility, severity, and mnemonic (e.g.,%LINK-3-UPDOWN.*down). - For
action cli commandto execute privileged commands,event manager session cli username [user]must be configured globally, and the referenced user must have privilege level 15. Without this, CLI actions silently fail or run with user-level privilege. - Action labels (1.0, 2.0, 3.0) determine execution order — actions run in ascending numerical order. Use decimal labels (1.5, 2.5) to insert actions between existing ones without renumbering. All actions within an applet run as a single atomic sequence unless
maxrunterminates the applet early. - Environment variables (
event manager environment [name] [value]) allow reusable parameters to be defined once and referenced in multiple applets with$_varnamesyntax. Change email addresses, thresholds, or server IPs without editing individual applets. - Always test new applets using
event noneandevent manager run [name]before attaching a live event trigger. This allows safe testing of all actions without waiting for a real fault event to occur. - The
ratelimit [seconds]option on the event clause prevents applet re-triggering for the specified duration after execution — essential for events that may recur rapidly (interface flapping, repeated syslog patterns) to prevent log flooding and flash exhaustion. - For CLI event applets,
sync no skip nomeans the applet runs asynchronously (does not block the operator's command) and the original command still executes.sync yes skip yesmeans the applet blocks the command and the original command is prevented from executing — use this pattern to block dangerous commands. show event manager policy registeredis the primary verification command — it lists all applets with their event type and pattern.show event manager history eventsshows execution history — if an expected event is missing from the history, the event detector pattern is not matching the actual syslog message.- On the exam: know the EEM applet structure (
event manager applet→event→action), the four most common event detectors (syslog, track, timer, cli), the key actions (syslog, cli command, mail), the purpose of environment variables, and thesync/skipoptions for CLI events. Also review debug commands and show logging as the primary tools for diagnosing why an EEM applet is or is not triggering.
event snmp
detector, see SNMP v2c
& v3 Configuration and
SNMP Traps.
For Python-based automation that can replace or supplement EEM for
more complex multi-device workflows, see
Python Netmiko Show
Commands.
For traffic capture at the time of an EEM-detected event, see
SPAN & RSPAN Port Mirroring.
For device hardening that complements EEM configuration-change
auditing, see
Login Security
& Brute-Force Protection.