Skip to content

Disaster Recovery and Resiliency

1. Overview

This document defines the disaster recovery (DR) and resiliency strategy for the Webex Contact Center deployment. It establishes Recovery Time Objectives (RTO), Recovery Point Objectives (RPO), failover procedures, and high-availability architecture to ensure business continuity during system failures or disasters.

Key Objectives: - Maintain 99.9% uptime for contact center services - Minimize data loss during failures (RPO) - Restore services rapidly after outages (RTO) - Provide graceful degradation during partial failures - Enable agent productivity during DR scenarios

2. Business Impact Analysis

2.1 Service Criticality Matrix

System Component Business Impact Downtime Cost (per hour) Criticality Level
Inbound call routing Revenue loss, customer dissatisfaction $50,000-100,000 🔴 Critical
Outbound calling Sales opportunity loss $25,000-50,000 🟠 High
Agent desktop Complete service disruption $50,000-100,000 🔴 Critical
CRM integration Reduced agent efficiency $10,000-25,000 🟡 Medium
Call recording Compliance risk $5,000-15,000 🟡 Medium
Historical reporting No immediate impact <$5,000 🟢 Low

2.2 Disaster Scenarios

Scenario Likelihood Impact Mitigation Strategy
CUBE/SBC hardware failure Medium High Active-standby HSRP pair
Internet circuit failure Medium Critical Dual ISP with BGP failover
Webex cloud regional outage Low Critical Multi-region routing (if available)
Data center power failure Low High UPS + generator, cloud failover
Cyber attack (DDoS) Medium High Cloud-based DDoS mitigation
Natural disaster (fire, flood) Low Critical Geographic redundancy
Mass agent connectivity loss Low High Agent re-login procedures

3. Recovery Objectives

3.1 RTO (Recovery Time Objective)

Definition: Maximum acceptable downtime before service must be restored.

Service RTO Target Justification
Inbound calls (PSTN → Webex) 5 minutes Critical revenue stream, automatic failover
Agent desktop (Webex App) 15 minutes Agent re-login, session restoration
CRM integration 1 hour Agents can operate without CRM temporarily
Reporting and analytics 24 hours Historical data, not time-sensitive
Call recording 4 hours Compliance requirement, buffer acceptable

Overall Target RTO: <15 minutes for core contact center services

3.2 RPO (Recovery Point Objective)

Definition: Maximum acceptable data loss (time delta between last backup and failure).

Data Type RPO Target Backup Method
Call Detail Records (CDR) 5 minutes Real-time replication to Webex cloud
Agent configurations 1 hour Sync to Webex Control Hub (cloud)
Call recordings 15 minutes Continuous upload to cloud storage
Historical reports 24 hours Daily database backups
CRM data 5 minutes Salesforce real-time sync

Overall Target RPO: <15 minutes for transactional data

4. High Availability Architecture

4.1 Component Redundancy Matrix

Component Primary Secondary/Backup Failover Method Failover Time
CUBE/SBC CUBE-Primary (10.50.1.10) CUBE-Standby (10.50.1.11) HSRP <30 seconds
Internet circuits ISP-A (500 Mbps) ISP-B (500 Mbps) BGP <30 seconds
Webex cloud US-East region US-West region (if multi-region) DNS failover 2-5 minutes
CUCM cluster Pub (10.10.10.10) Sub1/Sub2 (10.10.10.11/.12) Call Manager Group <10 seconds
Agent endpoints Office workstation Home laptop/VPN Agent switches devices 5-10 minutes
Power (data center) Utility power UPS → Generator Automatic 0 seconds (UPS)

4.2 High-Level Redundancy Diagram

┌──────────────────────────────────────────────────────────────────┐
│                    WEBEX CLOUD (Multi-Region)                     │
│  ┌───────────────────┐              ┌───────────────────┐        │
│  │  US-East Primary  │              │  US-West Backup   │        │
│  │  (Active)         │◄────────────►│  (Standby)        │        │
│  └─────────┬─────────┘              └─────────┬─────────┘        │
└────────────┼────────────────────────────────────┼────────────────┘
             │                                    │
        TLS/SRTP                             TLS/SRTP (failover)
             │                                    │
┌────────────▼────────────────────────────────────▼────────────────┐
│                       EDGE NETWORK (DMZ)                          │
│  ┌─────────────────┐                  ┌─────────────────┐        │
│  │ CUBE Primary    │    HSRP (VIP)    │ CUBE Standby    │        │
│  │ (10.50.1.10)    │◄────────────────►│ (10.50.1.11)    │        │
│  │ Priority: 110   │                  │ Priority: 100   │        │
│  └────────┬────────┘                  └────────┬────────┘        │
└───────────┼────────────────────────────────────┼─────────────────┘
            │                                    │
┌───────────▼────────────────────────────────────▼─────────────────┐
│                    INTERNAL FIREWALL                              │
└───────────┬───────────────────────────────────────────────────────┘
            │
┌───────────▼───────────────────────────────────────────────────────┐
│                    CORE NETWORK                                    │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐         │
│  │ CUCM Pub     │   │ CUCM Sub1    │   │ CUCM Sub2    │         │
│  │ (10.10.10.10)│   │ (10.10.10.11)│   │ (10.10.10.12)│         │
│  └──────────────┘   └──────────────┘   └──────────────┘         │
│           │                  │                  │                 │
│  ┌────────▼──────────────────▼──────────────────▼────┐           │
│  │             Agent Endpoints (IP Phones)           │           │
│  │             - Office: CUCM registered             │           │
│  │             - Home: Webex App (cloud)             │           │
│  └───────────────────────────────────────────────────┘           │
└───────────────────────────────────────────────────────────────────┘

5. CUBE/SBC Redundancy

5.1 Active-Standby HSRP Configuration

CUBE-Primary (Active):

interface GigabitEthernet0/0/1
 description Internal DMZ Interface
 ip address 10.50.1.10 255.255.255.0
 standby 1 ip 10.50.1.1
 standby 1 priority 110
 standby 1 preempt delay minimum 60
 standby 1 track 10 decrement 20

! Track internet reachability
track 10 ip sla 1 reachability

ip sla 1
 icmp-echo 8.8.8.8 source-interface GigabitEthernet0/0/0
 frequency 10
ip sla schedule 1 life forever start-time now

CUBE-Standby (Passive):

interface GigabitEthernet0/0/1
 description Internal DMZ Interface
 ip address 10.50.1.11 255.255.255.0
 standby 1 ip 10.50.1.1
 standby 1 priority 100
 standby 1 preempt delay minimum 60

5.2 CUBE Failover Scenarios

Scenario 1: CUBE Primary Hardware Failure

Failure: CUBE-Primary experiences hardware failure (CPU, memory, power supply).

Automatic Failover:

  1. T+0 seconds: HSRP hello messages stop from Primary
  2. T+3 seconds: Standby detects Primary failure (3× hello interval)
  3. T+5 seconds: Standby transitions to Active, assumes VIP 10.50.1.1
  4. T+10 seconds: Standby sends gratuitous ARP, updates switch MAC table
  5. T+30 seconds: All SIP traffic routing to new Active (Standby became Primary)

Impact: - 🔴 Active calls: Dropped (SIP sessions not replicated) - 🟢 New calls: Route to new Active CUBE immediately - 🟢 Agents: Automatically reconnect within 30 seconds

Expected Downtime: <30 seconds

Scenario 2: Internet Circuit Failure (Primary ISP)

Failure: ISP-A circuit fails (fiber cut, ISP outage).

Automatic Failover:

  1. T+0 seconds: IP SLA 1 fails to reach 8.8.8.8 (internet test)
  2. T+10 seconds: Track 10 marks as "down"
  3. T+11 seconds: HSRP priority decrements by 20 (110 → 90)
  4. T+12 seconds: Standby priority (100) > Active priority (90)
  5. T+15 seconds: Standby takes over as Active
  6. T+20 seconds: BGP reconverges to ISP-B circuit

Impact: - 🔴 Active calls: May experience 1-2 seconds of audio loss (packet buffering) - 🟢 New calls: Route via ISP-B immediately

Expected Downtime: <30 seconds

5.3 Manual Failover Procedure

Use Case: Planned maintenance on Primary CUBE.

Steps:

  1. Notify stakeholders: 30 minutes advance notice
  2. Verify Standby health: 
    show standby brief
show ip sla statistics
  3. Force failover: 
    ! On CUBE-Primary
interface GigabitEthernet0/0/1
 standby 1 priority 50
  4. Verify traffic flow: 
    ! On CUBE-Standby (now Active)
show call active voice brief
show ip nat translations
  5. Perform maintenance on former Primary
  6. Restore Primary: 
    ! On CUBE-Primary
interface GigabitEthernet0/0/1
 standby 1 priority 110
  7. Verify automatic preemption after 60 seconds

Expected Downtime: <10 seconds (planned, controlled)

6. Network Redundancy

6.1 Dual ISP Failover (BGP)

Architecture:

┌──────────────┐         ┌──────────────┐
│   ISP-A      │         │   ISP-B      │
│  (Primary)   │         │  (Backup)    │
│ AS 65100     │         │ AS 65200     │
└──────┬───────┘         └──────┬───────┘
       │                        │
       │ BGP Session            │ BGP Session
       │ Weight: 200            │ Weight: 100
       │                        │
┌──────▼────────────────────────▼───────┐
│         Border Router (BGP)            │
│         AS 65001 (Company)             │
└────────────────────────────────────────┘

BGP Configuration:

router bgp 65001
 bgp log-neighbor-changes
 neighbor 203.0.113.1 remote-as 65100
 neighbor 203.0.113.1 description ISP-A-Primary
 neighbor 203.0.114.1 remote-as 65200
 neighbor 203.0.114.1 description ISP-B-Backup
 !
 address-family ipv4
  network 203.0.113.0 mask 255.255.255.0
  neighbor 203.0.113.1 activate
  neighbor 203.0.113.1 weight 200
  neighbor 203.0.114.1 activate
  neighbor 203.0.114.1 weight 100
 exit-address-family

Failover Behavior:

  • Normal: Traffic uses ISP-A (higher weight)
  • ISP-A failure: BGP session drops, traffic automatically reroutes to ISP-B
  • Convergence time: <30 seconds

6.2 QoS During Failover

Ensure QoS policies applied to backup circuit:

interface GigabitEthernet0/0/1
 description ISP-B Backup Circuit
 service-policy output WAN-OUTBOUND-QOS

Critical: Voice traffic must be prioritized even during failover to prevent call quality degradation.

7. Webex Cloud Regional Redundancy

7.1 Multi-Region Architecture (If Available)

Note: Webex Contact Center multi-region redundancy is region-dependent. Verify with Cisco account team.

Primary Region: US-East (wxcc-us1.webex.com) Backup Region: US-West (wxcc-us2.webex.com) (if subscribed)

DNS-Based Failover:

; Primary A record
wxcc.yourcompany.com.   300   IN   A   64.100.1.10  (US-East)

; Backup A record (lower TTL for faster failover)
wxcc-dr.yourcompany.com. 60   IN   A   64.100.2.10  (US-West)

Failover Trigger:

  • Webex cloud health monitoring detects US-East region degradation
  • DNS automatically resolves to US-West backup region
  • Agents reconnect to backup region (automatic retry in Webex App)

Expected Failover Time: 2-5 minutes (DNS TTL + agent reconnection)

7.2 Single-Region Resilience

If multi-region not available:

Webex cloud operates with internal redundancy: - Load-balanced across multiple availability zones (AZs) - Auto-scaling for capacity - 99.99% SLA from Cisco

Dependency: Trust Cisco's cloud infrastructure resilience (no customer control).

8. Agent Re-Login and Recovery

8.1 Agent Failover Scenarios

Scenario A: CUBE Failover (Agent Unaffected)

Impact: Agents remain logged in, calls in progress may drop briefly.

Agent Action Required: None (automatic reconnection).

Expected Experience: - Active call: May experience 1-2 seconds of audio loss - New calls: Normal operation within 30 seconds

Scenario B: Webex Cloud Outage (Agents Disconnected)

Impact: All agents disconnected, Webex App shows "Reconnecting..."

Agent Action Required:

  1. Wait for automatic reconnection (2-3 minutes)
  2. Webex App retries connection automatically

  3. If multi-region available, redirects to backup region

  4. If reconnection fails, manual re-login:

  5. Close Webex App
  6. Reopen Webex App
  7. Click Contact Center widget
  8. Select Team → Go Available

Expected Downtime: 5-15 minutes (worst case)

Scenario C: Agent Internet Outage (Home Office)

Impact: Single agent cannot connect, others unaffected.

Agent Action Required:

  1. Switch to backup internet connection:
  2. Mobile hotspot (4G/5G tethering)

  3. Secondary ISP (if available)

  4. Or switch to backup device:

  5. Personal laptop/tablet with Webex App
  6. Call supervisor for reassignment

Expected Downtime: 10-30 minutes (depends on backup readiness)

8.2 Mass Agent Re-Login Procedure

Use Case: After datacenter failover or Webex cloud regional failover.

Steps:

  1. IT Operations sends notification:
  2. "Contact center services restored. Please log in to Webex."

  3. Email + Slack + SMS (multi-channel)

  4. Agents re-login in batches (avoid stampede):

  5. Batch 1 (0-5 min): Priority queues (Tier 1 support)
  6. Batch 2 (5-10 min): Standard queues

  7. Batch 3 (10-15 min): Back-office agents

  8. Monitor login rate:

  9. Target: <500 agents/minute (avoid overload)

  10. Use Webex Control Hub to monitor real-time logins

  11. Verify services:

  12. Supervisors test inbound call routing
  13. Sample agents verify screen pop (CRM integration)

Expected Recovery Time: 15-20 minutes to full operational capacity

9. Data Backup and Recovery

9.1 Webex Cloud Data (Managed by Cisco)

Call Detail Records (CDR): - Stored in Webex cloud (multi-AZ redundancy) - Retained per contract (typically 13 months) - Exportable via API or Control Hub

Agent Configurations: - Synchronized to Webex Control Hub (cloud) - Changes replicated in real-time - Version history available (30 days)

Call Recordings: - Uploaded to cloud storage (AWS S3 or Azure Blob) - Geo-redundant storage (3× replication) - Retention: Per compliance requirement (1-7 years)

Responsibility: Cisco (no customer action required)

9.2 On-Premises Data (Customer Managed)

CUBE/SBC Configurations:

Automated Backup:

archive
 log config
  logging enable
  notify syslog contenttype plaintext
 path ftp://10.10.10.50/cube-config-$h-$t
 write-memory
 time-period 1440

Backup Storage: - FTP server: 10.10.10.50 (on-premises) - Offsite replication: Sync to AWS S3 (nightly)

Retention: 90 days (30 days on-premises, 60 days offsite)

CUCM Cluster Backup:

Disaster Recovery System (DRS):

# Daily full backup
admin:utils disaster_recovery backup network

# Backup schedule
- Full backup: Daily at 2:00 AM
- Incremental: Not used (full only)
- Destination: SFTP server (10.10.10.51)

Backup Components: - CUCM database (devices, users, dial plan) - Unity Connection voicemail - IM&P (if deployed)

Retention: 14 days

9.3 CRM Data (Salesforce)

Backup Strategy:

  • Salesforce automatic daily backups (included in license)
  • Point-in-time recovery via Salesforce support
  • Nightly export to data warehouse (for analytics)

Responsibility: Salesforce (managed service)

10. Disaster Recovery Testing

10.1 DR Test Schedule

Test Type Frequency Scope Duration Success Criteria
CUBE failover Quarterly Primary → Standby 1 hour RTO <30 sec, 0% call loss
ISP failover Quarterly ISP-A → ISP-B 1 hour RTO <30 sec, QoS maintained
Agent re-login Bi-annually Mass logout/login 2 hours 100% agents online <15 min
Full datacenter failover Annually All systems 4 hours RTO <1 hour, RPO <15 min

10.2 DR Test Procedure (CUBE Failover)

Objective: Validate HSRP failover and call continuity.

Pre-Test:

  • Schedule during off-peak hours (e.g., Saturday 10 AM)
  • Notify all stakeholders (agents, supervisors, IT, business)
  • Prepare rollback plan

Test Steps:

  1. T-10 min: Verify baseline (all systems green) 

    show standby brief
show call active voice brief

  2. T+0 min: Initiate failover 

    ! On CUBE-Primary
interface GigabitEthernet0/0/1
 shutdown

  3. T+1 min: Verify Standby becomes Active 

    ! On CUBE-Standby
show standby brief
! Should show: Active, VIP 10.50.1.1

  4. T+2 min: Place test call (inbound and outbound)

  5. Verify call completes successfully

  6. Check audio quality (no dropouts)

  7. T+10 min: Restore Primary 

    ! On CUBE-Primary
interface GigabitEthernet0/0/1
 no shutdown

  8. T+15 min: Verify Primary resumes Active role (preemption)

Success Criteria:

  • ✅ Failover time: <30 seconds
  • ✅ Test calls: 100% success rate
  • ✅ Audio quality: No degradation
  • ✅ Agents: No manual intervention required

Post-Test:

  • Document results in test report
  • Review any issues with architecture team
  • Update DR runbook if needed

11. Incident Response and Escalation

11.1 Incident Severity Definitions

Severity Definition Response Time Escalation Path
P1 (Critical) Complete service outage 5 minutes NOC → Manager → VP IT
P2 (High) Partial outage (>25% agents affected) 15 minutes NOC → Manager
P3 (Medium) Degraded performance or <25% affected 1 hour NOC → Supervisor
P4 (Low) Cosmetic issue, no service impact 4 hours NOC (no escalation)

11.2 Incident Response Flow

┌─────────────────┐
│ Incident Occurs │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Monitoring      │ (Automatic detection via alerts)
│ Detects Issue   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ NOC Receives    │ (ServiceNow ticket auto-created)
│ Alert           │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ NOC Classifies  │ (Assigns severity: P1/P2/P3/P4)
│ Severity        │
└────────┬────────┘
         │
    ┌────▼────┐
    │ P1/P2?  │
    └────┬────┘
         │
    ┌────▼─────────┐      ┌──────────────┐
    │ YES          │      │ NO (P3/P4)   │
    │ Escalate     │      │ Follow       │
    │ Immediately  │      │ Standard SLA │
    └────┬─────────┘      └──────────────┘
         │
         ▼
┌─────────────────┐
│ Engage Vendors  │ (Cisco TAC, carrier, etc.)
│ if needed       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Execute DR Plan │ (This document)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Service Restored│
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Post-Incident   │ (Root cause analysis within 48 hours)
│ Review (PIR)    │
└─────────────────┘

11.3 On-Call Rotation

24×7 On-Call Coverage:

Role Primary Secondary Tertiary
Network Engineer John Doe Jane Smith Bob Wilson
Telephony Engineer Alice Johnson Charlie Brown Diana Prince
Webex Admin Eve Martinez Frank Castle Grace Hopper
Manager (Escalation) Henry Kim Isabel Ramirez —

On-Call Schedule: Rotating weekly shifts (Monday 8 AM → Monday 8 AM)

Escalation SLA:

  • P1: Response within 5 minutes
  • P2: Response within 15 minutes
  • P3: Response within 1 hour

12. Business Continuity Plan

12.1 Complete Datacenter Failure

Scenario: On-premises datacenter becomes unavailable (fire, flood, extended power outage).

Impact: - CUBE/SBC unavailable → PSTN calls cannot reach Webex - CUCM unavailable → IP phones cannot register - Agents with Webex App unaffected (cloud-based)

Recovery Strategy:

Immediate (0-1 hour):

  1. Activate PSTN failover routing:
  2. Contact carrier: Reroute DIDs to backup SIP trunk (cloud-based SBC)

  3. Or activate cloud-connected PSTN (if pre-configured)

  4. Agents switch to Webex App (cloud):

  5. Home agents continue working (unaffected)
  6. Office agents switch to laptops with Webex App

Short-term (1-4 hours):

  1. Deploy temporary CUBE in AWS/Azure:
  2. Spin up virtual CUBE instance (Cisco CSR 1000V)
  3. Configure SIP trunks to carrier and Webex

  4. Update firewall rules to allow traffic

  5. Redirect CUCM-registered phones (optional):

  6. If critical, provision Webex Calling licenses
  7. Re-register phones to Webex cloud

Long-term (1-7 days):

  1. Restore datacenter or migrate to permanent cloud architecture

12.2 Pandemic or Mass Work-From-Home

Scenario: Sudden requirement for 100% remote agents (e.g., COVID-19).

Preparation:

  • All agents have Webex App installed on personal devices
  • VPN capacity scaled to support 100% remote workforce
  • Split-tunnel VPN configured (Webex direct to cloud)
  • Headsets shipped to agent homes (or agents use personal)

Activation:

  1. Day 0: Announce work-from-home policy
  2. Day 1: Agents log in from home via Webex App
  3. Day 2: IT support for connectivity issues (helpdesk surge staffing)
  4. Day 3+: Normal operations (remote)

Expected Service Continuity: 95%+ (small dip due to home network issues)

13. Runbook Summary

13.1 Quick Reference: Common Failures

Failure Detection Response Estimated Downtime
CUBE Primary failure HSRP alert Automatic failover to Standby <30 seconds
Internet circuit failure BGP down alert Automatic failover to ISP-B <30 seconds
Webex cloud outage Agent reports, monitoring Wait for Cisco resolution or multi-region failover 2-15 minutes
CUCM failure Call Manager down alert Automatic failover to Sub1/Sub2 <10 seconds
Mass agent disconnect Dashboard shows 0 agents online Investigate cause, send re-login instructions 15-30 minutes

13.2 Emergency Contacts

Role Name Phone Email
IT Manager (Escalation Point) Henry Kim +1-XX5-0100 hkim@company.com
Network Engineer (On-Call) John Doe +1-XX5-0101 jdoe@company.com
Telephony Engineer (On-Call) Alice Johnson +1-XX5-0102 ajohnson@company.com
Cisco TAC (Support) — +1-800-553-2447 —
ISP-A (AT&T Support) — +1-800-288-2020 —
ISP-B (Verizon Support) — +1-800-837-4966 —
Webex Control Hub Escalation — +1-844-630-4635 webex-support@cisco.com

14. Continuous Improvement

14.1 Post-Incident Review (PIR) Process

Trigger: Any P1 or P2 incident.

Timeline: Within 48 hours of resolution.

PIR Template:

```markdown

Post-Incident Review

Incident ID: INC-2025-001 Date/Time: 2025-11-15 14:30 UTC Severity: P1 (Critical) Duration: 12 minutes

Summary

[Brief description of what happened]

Timeline

  • 14:30: Incident detected (monitoring alert)
  • 14:32: NOC acknowledged, began investigation
  • 14:35: Root cause identified (CUBE primary failure)
  • 14:37: Failover to CUBE standby completed
  • 14:42: Service fully restored, all agents online

Root Cause

[Technical explanation]

Impact

  • Affected users: 1,000 agents
  • Call loss: 15 active calls dropped
  • Customer impact: High (wait times increased)

Actions Taken

  1. CUBE standby promoted to active (automatic)
  2. Network team verified failover success
  3. Agents reconnected automatically

Lessons Learned

  • Failover worked as designed
  • HSRP preemption delay (60s) appropriate
  • Agent training needed on automatic reconnection

Action Items

  1. Replace failed CUBE hardware (RMA initiated)
  2. Review CUBE monitoring (add predictive alerts)
  3. Agent FAQ updated (what to expect during failover)

14.2 Quarterly DR Review

Agenda:

  1. Review RTO/RPO metrics (did we meet targets?)
  2. Update disaster scenarios (new threats?)
  3. Validate contact information (on-call rotation current?)
  4. Review test results (any failures?)
  5. Update DR documentation (this document)

Attendees: - IT Manager - Network Engineer - Telephony Engineer - Webex Admin - Business Continuity Manager