> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pensionsportal.ie/llms.txt
> Use this file to discover all available pages before exploring further.
# Incident Response Runbook
> Step-by-step incident response procedures for PensionsPortal.ie
## Overview
This runbook covers incident response for PensionsPortal.ie, an IORP II compliance platform handling GDPR-protected personal data (including AES-256 encrypted PPS numbers) for Irish pension trustees. Incidents must be handled in compliance with **GDPR Art. 33** (72-hour breach notification), **DORA Art. 17** (ICT incident reporting), and **DORA Art. 13** (post-incident review).
This runbook must be reviewed and contacts updated before the platform goes live. All email addresses below are placeholders. Store the real contacts in the company password manager and link to this document from the on-call channel.
***
## 1. Incident Classification Matrix
| Severity | Criteria | Response Time | Escalation |
| ----------------- | --------------------------------------------------------------------- | ----------------- | ----------------------------- |
| **P0 — Critical** | Data breach, system fully down, unauthorized data access | 15 minutes | CTO + DPO + Legal immediately |
| **P1 — High** | Auth bypass, production DB unreachable, DORA-reportable ICT incident | 1 hour | Engineering Lead + CTO |
| **P2 — Medium** | Significant performance degradation, failed email delivery, AI outage | 4 hours | On-call engineer |
| **P3 — Low** | Minor UI bug, non-critical feature outage | Next business day | Normal ticket process |
When in doubt, escalate up. Downgrading a P0 to a P1 after assessment is always safer than under-escalating a genuine data breach. GDPR Art. 33 clock starts from when you "become aware" — not from when you finish your assessment.
***
## 2. On-Call Contacts
| Role | Contact | Notes |
| -------------------------- | --------------------------------------------------------------------------- | --------------------------------------------- |
| Security / DPO | [dpo@pensionsportal.ie](mailto:dpo@pensionsportal.ie) | Required for any P0 involving personal data |
| CTO | [cto@pensionsportal.ie](mailto:cto@pensionsportal.ie) | Required for P0 and P1 |
| Engineering | [engineering@pensionsportal.ie](mailto:engineering@pensionsportal.ie) | On-call for all severity levels |
| Pensions Authority (DORA) | [supervision@pensionsauthority.ie](mailto:supervision@pensionsauthority.ie) | DORA Art. 17 significant ICT incident reports |
| Data Protection Commission | dpc.ie/contact | GDPR Art. 33 breach notifications |
These are placeholder contacts. Replace with real names, phone numbers, and out-of-hours contact methods before production launch. Store actual contact details in the company password manager, not in this document.
***
## 3. P0 — Data Breach Response Procedure
Follow these steps sequentially. Time targets are maximums — act faster where possible.
### Step 1 — Detect
An incident may be detected via:
* Automated monitoring alerts (Vercel, Neon, Cloudflare)
* User or broker report
* Internal team discovery
* Third-party security researcher notification
**Immediately** open an incident channel (e.g., `#incident-YYYY-MM-DD`) and assign an Incident Commander.
### Step 2 — Contain (within 15 minutes)
Act to stop the bleeding before completing the full assessment.
**If an active external attack or breach is suspected:**
```bash theme={null}
# 1. Block attacker IP in Vercel Dashboard
# Vercel Dashboard → Settings → Firewall → Add Rule → Block IP
# 2. Enable Cloudflare Under Attack Mode
# Cloudflare Dashboard → Security → Settings → Security Level → Under Attack
# 3. If DB compromise suspected, contact Neon support immediately
# https://neon.tech/docs/introduction/support
# Request: suspend external access to project
```
**If unauthorized internal access is suspected:**
```sql theme={null}
-- Disable the compromised user account
UPDATE users SET role = 'Disabled' WHERE id = '';
```
Log every containment action taken with timestamps in the incident channel.
### Step 3 — Assess (within 1 hour)
Query audit logs to identify the scope of suspicious activity:
```sql theme={null}
-- Review all activity in the past 24 hours
SELECT
al.id,
al.actor_id,
u.email AS actor_email,
al.action,
al.entity_type,
al.entity_id,
al.ip_address,
al.user_agent,
al.timestamp
FROM audit_logs al
LEFT JOIN users u ON u.id = al.actor_id
WHERE al.timestamp > NOW() - INTERVAL '24 hours'
ORDER BY al.timestamp DESC;
-- Look specifically for PPS decryptions
SELECT * FROM audit_logs
WHERE action LIKE '%pps%' OR action LIKE '%decrypt%'
AND timestamp > NOW() - INTERVAL '24 hours';
```
Complete the following assessment:
* [ ] Identify the attack vector (how did the breach occur?)
* [ ] Identify affected data categories (PII? PPS numbers? Compliance records? Documents?)
* [ ] Estimate the number of affected data subjects (members, users)
* [ ] Identify which tenants (brokers) are affected
* [ ] Determine whether PPS encryption key may be compromised
* [ ] Determine whether AUTH\_SECRET may be compromised
### Step 4 — Notify (within 72 hours — GDPR Art. 33)
The 72-hour clock runs from the moment you "become aware" of a breach involving personal data — not from when the investigation concludes. File the initial DPC notification even if the full scope is not yet known. You can supplement with additional information later.
**Data Protection Commission (DPC) — GDPR Art. 33:**
* Notify at [dpc.ie](https://www.dataprotection.ie) if any personal data is involved
* Use the DPC's online breach notification form
* Include: incident date/time, data categories affected, estimated number of subjects, containment actions taken, remediation plan
**Pensions Authority — DORA Art. 17:**
* Notify if the incident qualifies as a "significant ICT incident" (see Section 4 below)
* Email: [supervision@pensionsauthority.ie](mailto:supervision@pensionsauthority.ie)
* Subject line: `DORA Incident Notification — PensionsPortal.ie — [YYYY-MM-DD]`
**Affected Brokers / Trustees:**
* Notify affected broker firms via their registered contact email
* Use the communication template in Section 7
**Notification template:**
```
Incident Date/Time: [UTC timestamp]
Data Categories Affected: [e.g., member names, email addresses, encrypted PPS numbers]
Estimated Number of Affected Data Subjects: [count]
Affected Pension Schemes: [scheme names or "under investigation"]
Containment Actions Taken: [list actions with timestamps]
Remediation Plan: [steps and expected completion dates]
DPC Notification Reference: [reference number once filed]
```
### Step 5 — Remediate
```bash theme={null}
# If AUTH_SECRET is compromised (forces all sessions to invalidate)
# 1. Generate a new AUTH_SECRET
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
# 2. Update in Vercel Dashboard
# Vercel Dashboard → Project → Settings → Environment Variables → AUTH_SECRET
# 3. Redeploy to apply new secret
vercel --prod
```
If `PPS_ENCRYPTION_KEY` may be compromised, follow the **PPS Encryption Key Rotation Procedure** in Section 5 before anything else.
Additional remediation steps:
* [ ] Patch the identified vulnerability
* [ ] Reset credentials for any compromised accounts
* [ ] Review and tighten Cloudflare WAF rules
* [ ] Run full audit log review for the affected period
* [ ] Verify all containment measures are still active
* [ ] Re-enable normal operations only after remediation is confirmed
### Step 6 — Post-Incident Review (within 5 business days — DORA Art. 13)
Write a post-mortem document covering:
| Section | Content |
| ----------- | ------------------------------------------------------------------- |
| Timeline | Chronological sequence of events from first indicator to resolution |
| Root Cause | Technical and process root cause analysis |
| Impact | Data subjects affected, business impact, regulatory exposure |
| Containment | Actions taken and their effectiveness |
| Remediation | Changes made to prevent recurrence |
| Prevention | Recommended longer-term security improvements |
File the completed post-mortem with the DPO for **GDPR Art. 30** Records of Processing Activities.
***
## 4. DORA ICT Incident Reporting
Per **DORA Art. 17**, significant ICT incidents affecting pension scheme operations must be reported to the Pensions Authority.
### Classification as "Significant"
An ICT incident is significant if it:
* Causes service unavailability affecting trustees' ability to meet compliance obligations
* Results in data loss or corruption affecting pension records
* Involves a security breach affecting member PII or scheme data
* Disrupts critical functions for more than a defined threshold period
### Reporting Timeline
| Stage | Deadline | Content |
| -------------------- | ------------------------------------------- | ---------------------------------------------------------------------------- |
| Initial notification | 4 hours after classification as significant | Incident description, systems affected, immediate business impact |
| Intermediate report | 72 hours | Updated assessment, containment measures, estimated resolution |
| Final report | 1 month | Root cause, full impact assessment, remediation actions, prevention measures |
### DORA Notification Email Format
Send to: [supervision@pensionsauthority.ie](mailto:supervision@pensionsauthority.ie)
```
Subject: DORA Incident Notification — PensionsPortal.ie — [YYYY-MM-DD]
Incident Reference: [internal incident ID]
Date/Time of Detection: [UTC]
Date/Time of Occurrence (estimated): [UTC]
Systems Affected: [e.g., Production database, Authentication service]
Business Impact: [describe effect on pension scheme operations]
Containment Actions Taken: [list with timestamps]
Estimated Resolution Date: [date]
Contact for Follow-Up: [name, email, phone]
```
***
## 5. PPS Encryption Key Rotation Procedure
PPS key rotation is a destructive operation. Every PPS number in the database must be decrypted with the old key and re-encrypted with the new key. Perform this during a maintenance window with the application in read-only mode. A failed rotation can result in permanent data loss.
Follow these steps in order:
**Step 1 — Generate new encryption key:**
```bash theme={null}
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
# Output: a 64-character hex string — this is your new PPS_ENCRYPTION_KEY
```
**Step 2 — Add new key to Vercel environment (do not remove old key yet):**
```bash theme={null}
# In Vercel Dashboard → Project → Settings → Environment Variables
# Add: PPS_ENCRYPTION_KEY_NEW =
# Keep: PPS_ENCRYPTION_KEY = (do not remove)
```
**Step 3 — Run key rotation migration:**
```bash theme={null}
# Requires both PPS_ENCRYPTION_KEY (old) and PPS_ENCRYPTION_KEY_NEW (new) in environment
npm run db:rotate-pps-key
```
**Pre-launch requirement:** The `db:rotate-pps-key` script must be implemented before the platform processes real PPS numbers. The script must: (1) select all members with a non-null `pps_number_encrypted`, (2) decrypt each with `PPS_ENCRYPTION_KEY`, (3) re-encrypt with `PPS_ENCRYPTION_KEY_NEW`, (4) update the record in a transaction, (5) log each rotation in `audit_logs`, (6) rollback on any error. Test this procedure end-to-end in staging before production launch.
**Step 4 — Verify re-encryption:**
```sql theme={null}
-- Confirm no members still have unrotated PPS data
-- (Script should log a rotation record per member — count should match)
SELECT COUNT(*) FROM audit_logs
WHERE action = 'PPS_KEY_ROTATED'
AND timestamp > NOW() - INTERVAL '1 hour';
-- Compare against total members with PPS numbers
SELECT COUNT(*) FROM members
WHERE pps_number_encrypted IS NOT NULL;
```
**Step 5 — Swap keys in Vercel environment:**
```bash theme={null}
# In Vercel Dashboard → Project → Settings → Environment Variables
# 1. Remove: PPS_ENCRYPTION_KEY (old)
# 2. Rename: PPS_ENCRYPTION_KEY_NEW → PPS_ENCRYPTION_KEY
```
**Step 6 — Redeploy and verify:**
```bash theme={null}
vercel --prod
```
After deployment:
* [ ] Log in as a BrokerAdmin and view a member with a PPS number
* [ ] Confirm PPS decrypts and displays correctly
* [ ] Confirm the decryption is logged in `audit_logs`
* [ ] Log key rotation completion in `audit_logs` with actor `system`
***
## 6. Rollback Procedure
For application rollback steps (reverting to a previous deployment), see [Vercel Deployment Runbook](/deployment/vercel).
For database rollback (restoring to a pre-incident state), see [Backup & Restore Runbook](/runbooks/backup-restore).
***
## 7. Customer Communication Template
Use this template for communicating with broker firms and trustees during incidents. Send via email to the broker's registered contact address.
```
Subject: PensionsPortal.ie Service Notice — [YYYY-MM-DD]
Dear [Broker Firm Name],
We are investigating a technical issue affecting [feature/service description].
Current status: [investigating / contained / resolved]
Estimated resolution: [time estimate or "under investigation"]
We will provide a further update by [specific time].
Your data security is our priority. [If data breach: We are treating this as
a potential data protection incident and our Data Protection Officer has been
notified. We will contact you directly if your data or your clients' data
has been affected.]
If you have urgent compliance deadlines that cannot be met due to this issue,
please contact support@pensionsportal.ie and we will prioritise your case.
Regards,
PensionsPortal.ie Engineering Team
```
For confirmed data breaches involving a broker's client data, the DPO must review and approve all external communications before they are sent. Do not send breach-related communications without DPO sign-off.