# Incident Response Plan

## Lilith Platform Smart Contract Security

### Version 1.0 | Last Updated: 2024

---

## Table of Contents

1. [Incident Severity Classification](#incident-severity-classification)
2. [Response Team & Roles](#response-team--roles)
3. [Detection & Alerting](#detection--alerting)
4. [Response Procedures](#response-procedures)
5. [Communication Protocols](#communication-protocols)
6. [Post-Incident Procedures](#post-incident-procedures)
7. [User Compensation Process](#user-compensation-process)
8. [Tabletop Exercises](#tabletop-exercises)
9. [On-Call Rotation](#on-call-rotation)

---

## Incident Severity Classification

### P0 - CRITICAL (Active Exploit)

**Definition:** Active exploitation in progress, funds at immediate risk, or complete system compromise.

**Examples:**
- Ongoing fund drain from contracts
- Private key compromise
- Smart contract logic exploit being actively used
- Flash loan attack successfully bypassing guards
- Unauthorized ownership transfer

**Response Time:** IMMEDIATE (< 5 minutes)

**Actions:**
1. Emergency pause ALL affected contracts
2. Alert entire security team via emergency channel
3. Initiate war room immediately
4. Notify legal team
5. Prepare public communication

**Escalation:** CEO, CTO, Legal, PR within 15 minutes

---

### P1 - HIGH (Vulnerability Discovered)

**Definition:** Critical vulnerability identified but not yet exploited. Immediate mitigation required.

**Examples:**
- Critical vulnerability found in audit or bug bounty
- Potential exploit vector identified in production
- Suspicious transaction patterns detected
- Failed flash loan attack (guard triggered)
- Unauthorized role changes detected

**Response Time:** < 1 hour

**Actions:**
1. Assess exploitability and impact
2. Consider preventive pause if high risk
3. Begin mitigation development
4. Monitor for exploitation attempts
5. Prepare patch deployment

**Escalation:** Security Lead, Smart Contract Team Lead within 30 minutes

---

### P2 - MEDIUM (Issue Identified)

**Definition:** Non-critical issue or vulnerability with no immediate risk. Scheduled mitigation.

**Examples:**
- Medium-severity vulnerability in non-critical path
- Gas optimization issues causing failed transactions
- UI/frontend issues affecting user experience
- Suboptimal contract configuration

**Response Time:** < 24 hours (next business day)

**Actions:**
1. Document issue thoroughly
2. Assess priority for next release
3. Develop and test fix
4. Schedule deployment

**Escalation:** Team Lead within 4 hours

---

### P3 - LOW (Minor Issue)

**Definition:** Minor issues, optimizations, or improvements. No security impact.

**Examples:**
- Documentation updates needed
- Minor UI improvements
- Non-critical gas optimizations
- Feature enhancement requests

**Response Time:** Next sprint/release cycle

**Actions:**
1. Create tracking ticket
2. Prioritize in backlog
3. Address in regular development cycle

---

## Response Team & Roles

### Incident Commander (IC)
- **Responsibility:** Overall coordination of incident response
- **Authority:** Can authorize emergency actions including pause
- **Primary:** Security Lead
- **Backup:** CTO

### Technical Lead
- **Responsibility:** Technical analysis and mitigation
- **Authority:** Direct smart contract and infrastructure changes
- **Primary:** Smart Contract Team Lead
- **Backup:** Senior Smart Contract Engineer

### Communications Lead
- **Responsibility:** Internal and external communications
- **Authority:** Approve and publish public statements
- **Primary:** PR/Communications Manager
- **Backup:** CEO

### Operations Lead
- **Responsibility:** Monitor systems and execute commands
- **Authority:** Execute authorized commands on infrastructure
- **Primary:** DevOps Lead
- **Backup:** Senior DevOps Engineer

---

## Detection & Alerting

### Monitoring Systems

| System | Coverage | Alert Destination |
|--------|----------|-------------------|
| OpenZeppelin Defender | On-chain events | PagerDuty, Slack |
| Forta | Suspicious patterns | PagerDuty, Slack |
| The Graph | Historical analysis | Grafana, Email |
| Internal Monitor | Real-time events | Slack #alerts |

### Alert Priority Mapping

| Severity | PagerDuty | Slack | Email | SMS |
|----------|-----------|-------|-------|-----|
| P0 | Immediate | #incidents | Yes | Yes |
| P1 | High | #alerts | Yes | On-call |
| P2 | Low | #monitoring | Yes | No |
| P3 | None | #monitoring | No | No |

---

## Response Procedures

### P0 - Active Exploit Response

```
TIME    ACTION
0:00    Alert received
0:02    Incident Commander confirms P0 severity
0:05    Emergency pause executed on all affected contracts
0:10    War room opened (video call link sent)
0:15    Initial assessment complete
0:30    Stakeholders briefed
1:00    Public holding statement published
2:00    Detailed analysis complete
4:00    Remediation plan drafted
24:00   Post-mortem meeting scheduled
```

#### Emergency Pause Authorization

The following roles can authorize emergency pause:

1. **CEO** - Full authority
2. **CTO** - Full authority
3. **Security Lead** - Full authority for P0/P1
4. **Smart Contract Lead** - Authority for P0 only
5. **On-Call Engineer** - Authority for P0 only (with immediate escalation)

#### Emergency Pause Procedure

```bash
# 1. Connect to production network
export NETWORK=mainnet

# 2. Execute emergency pause via Safe
# Option A: Using Defender Relay
npx hardhat run scripts/emergency-pause.ts --network $NETWORK

# Option B: Direct Safe transaction
# Submit pause transaction to Safe multisig
# Collect required signatures
# Execute transaction

# 3. Verify pause
npx hardhat run scripts/verify-pause.ts --network $NETWORK

# 4. Notify all stakeholders
# Send to: #incidents, leadership, legal
```

### P1 - Vulnerability Response

```
TIME    ACTION
0:00    Alert/report received
0:15    Initial triage by on-call
0:30    Severity confirmed by Security Lead
1:00    Assessment meeting with relevant engineers
2:00    Mitigation strategy decided
4:00    Mitigation developed (if applicable)
8:00    Mitigation tested on testnet
24:00   Fix deployed or preventive measures in place
48:00   Full resolution deployed
72:00   Post-mortem complete
```

---

## Communication Protocols

### Internal Channels

| Channel | Purpose | Audience |
|---------|---------|----------|
| #incidents | Active P0/P1 coordination | Response team |
| #alerts | All security alerts | Engineering |
| #monitoring | Routine monitoring | DevOps |
| War Room (Video) | Real-time coordination | Response team |

### External Channels

| Channel | Purpose | Timing |
|---------|---------|--------|
| Twitter/X | User updates | Within 1 hour of P0 |
| Discord | Community updates | Within 2 hours |
| Blog | Detailed updates | Within 24 hours |
| Email | Direct user notification | As needed |

### Communication Templates

#### Holding Statement (P0)

```
[SECURITY NOTICE]

We are currently investigating a security incident affecting [CONTRACT/SERVICE].

For your protection, we have temporarily paused [affected services].

Your funds are [safe/being assessed]. We will provide updates every [30 minutes/hour].

Status page: [link]
Discord: [link]

We appreciate your patience.
```

#### Resolution Statement

```
[SECURITY UPDATE]

The security incident reported on [DATE] has been resolved.

Summary:
- Issue: [Brief description]
- Impact: [Number of affected users, funds at risk]
- Resolution: [Actions taken]
- Timeline: [Key events]

Affected users will be [compensated/contacted] within [timeframe].

Detailed post-mortem: [link]
```

---

## Post-Incident Procedures

### Post-Mortem Template

```markdown
# Incident Post-Mortem: [INCIDENT-ID]

## Summary
- **Incident Date:** YYYY-MM-DD
- **Severity:** P0/P1/P2
- **Duration:** X hours
- **Impact:** [Description]

## Timeline
| Time (UTC) | Event |
|------------|-------|
| HH:MM | Alert received |
| HH:MM | Incident confirmed |
| HH:MM | Mitigation started |
| HH:MM | Resolution deployed |
| HH:MM | All clear declared |

## Root Cause
[Detailed technical explanation]

## Impact Assessment
- Users affected: X
- Funds at risk: X ETH
- Actual losses: X ETH
- Transactions affected: X

## Response Evaluation
### What Went Well
- [Item 1]
- [Item 2]

### What Could Improve
- [Item 1]
- [Item 2]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Action] | [Name] | [Date] | [Status] |

## Lessons Learned
[Key takeaways for future prevention]
```

### Post-Mortem Meeting Agenda

1. Timeline review (15 min)
2. Root cause analysis (20 min)
3. Response effectiveness (15 min)
4. Prevention measures (20 min)
5. Action item assignment (10 min)

---

## User Compensation Process

### Eligibility Assessment

1. **Verify affected users from on-chain data**
   - Query affected transactions
   - Calculate individual losses
   - Verify wallet ownership

2. **Compensation Categories**

| Category | Compensation | Timeline |
|----------|--------------|----------|
| Direct fund loss | 100% of verified loss | 7 days |
| Failed transaction fees | 100% of gas spent | 7 days |
| Opportunity cost | Case by case | 14 days |
| Goodwill | Discretionary | 30 days |

3. **Compensation Mechanisms**

- **On-chain refund:** Direct transfer to affected wallets
- **Claim portal:** User-initiated claims with verification
- **Airdrop:** For smaller amounts to many users

### Claim Verification

```solidity
// Claims require:
// 1. Proof of affected transaction
// 2. Wallet signature verification
// 3. Amount calculation verification
// 4. Duplicate claim prevention
```

---

## Tabletop Exercises

### Quarterly Exercise Schedule

| Quarter | Scenario | Focus Area |
|---------|----------|------------|
| Q1 | Flash loan attack | Detection & pause |
| Q2 | Private key compromise | Multi-sig recovery |
| Q3 | Smart contract exploit | Patching & deployment |
| Q4 | Full incident simulation | End-to-end response |

### Exercise Format

1. **Preparation (1 week before)**
   - Define scenario
   - Prepare injects
   - Brief exercise controller

2. **Execution (2-3 hours)**
   - Scenario introduction (15 min)
   - Response simulation (90 min)
   - Debrief (45 min)

3. **Follow-up (1 week after)**
   - Document lessons learned
   - Update procedures
   - Track action items

### Sample Scenarios

#### Scenario 1: Flash Loan Attack
```
INJECT 1: Forta bot detects unusual transaction pattern
INJECT 2: User reports unexpected fund movement
INJECT 3: Attacker drains 50 ETH from AccessToken
INJECT 4: Media inquiry received
```

#### Scenario 2: Key Compromise
```
INJECT 1: Unusual Safe transaction submitted
INJECT 2: Team member reports phishing attempt
INJECT 3: Unauthorized ownership transfer detected
INJECT 4: Attacker posts on social media
```

---

## On-Call Rotation

### Schedule

- **Primary:** 1 week rotation
- **Secondary:** 1 week rotation (shadow)
- **Coverage:** 24/7/365
- **Handoff:** Monday 10:00 UTC

### On-Call Responsibilities

1. Monitor #alerts channel
2. Respond to PagerDuty within 15 minutes
3. Escalate P0/P1 immediately
4. Document all incidents
5. Brief next on-call at handoff

### Escalation Path

```
On-Call Engineer (15 min)
    ↓
Security Lead (30 min)
    ↓
CTO (1 hour)
    ↓
CEO (as needed)
```

### On-Call Tools

- [ ] PagerDuty mobile app configured
- [ ] Safe wallet access (view only)
- [ ] Defender dashboard access
- [ ] Slack mobile with #incidents notifications
- [ ] War room video call link
- [ ] Emergency contact list

---

## Appendices

### A. Emergency Contact List

| Role | Primary | Secondary |
|------|---------|-----------|
| CEO | [REDACTED] | [REDACTED] |
| CTO | [REDACTED] | [REDACTED] |
| Security Lead | [REDACTED] | [REDACTED] |
| Legal Counsel | [REDACTED] | [REDACTED] |
| PR Lead | [REDACTED] | [REDACTED] |

### B. External Contacts

| Organization | Contact | Purpose |
|--------------|---------|---------|
| Trail of Bits | [REDACTED] | Emergency audit |
| Chainalysis | [REDACTED] | Fund tracing |
| OpenZeppelin | [REDACTED] | Technical support |
| Legal Firm | [REDACTED] | Legal counsel |

### C. Quick Reference Card

```
P0 RESPONSE CHECKLIST
□ Confirm severity with IC
□ Execute emergency pause
□ Open war room
□ Notify stakeholders
□ Prepare holding statement
□ Begin root cause analysis
□ Document timeline
□ Prepare user communication
```

---

*This document is reviewed and updated quarterly. Last review: [DATE]*
