stanta/homelab
1
0
Fork 0
Code Issues 30 Pull requests Projects Releases Packages Wiki Activity Actions

Run the disruptive monitoring failure drills #18

New issue
Open
opened 2026-08-23 17:07:54 +00:00 by stanta · 0 comments
stanta commented 2026-08-23 17:07:54 +00:00
Owner
Copy link

Outcome

Complete the remaining service-stop, host-stop, DNS, uplink, notification, and restore drills in a maintenance window.

Current state

Non-disruptive alert lifecycle tests passed. Drills that interrupt live infrastructure remain untested.

Work

  • Agree on a maintenance window and recovery order.
  • Test host and exporter failures.
  • Test backend and Traefik failures with inhibition.
  • Test DNS and home uplink failures.
  • Test Prometheus, Alertmanager, Alerta, and ntfy failures.
  • Test a real maintenance silence.
  • Restore monitoring state into a clean target.

Validation

  • Every drill records firing delay, notification, acknowledgement, inhibition, recovery, and resolved delivery.
  • Each service is restored before the next drill.
  • Monitoring-plane failure is detected outside its own failure domain.

References

  • iac/MONITORING-ALERTING-PLAN.md
  • scripts/validate-monitoring.py
## Outcome Complete the remaining service-stop, host-stop, DNS, uplink, notification, and restore drills in a maintenance window. ## Current state Non-disruptive alert lifecycle tests passed. Drills that interrupt live infrastructure remain untested. ## Work - [ ] Agree on a maintenance window and recovery order. - [ ] Test host and exporter failures. - [ ] Test backend and Traefik failures with inhibition. - [ ] Test DNS and home uplink failures. - [ ] Test Prometheus, Alertmanager, Alerta, and ntfy failures. - [ ] Test a real maintenance silence. - [ ] Restore monitoring state into a clean target. ## Validation - [ ] Every drill records firing delay, notification, acknowledgement, inhibition, recovery, and resolved delivery. - [ ] Each service is restored before the next drill. - [ ] Monitoring-plane failure is detected outside its own failure domain. ## References - `iac/MONITORING-ALERTING-PLAN.md` - `scripts/validate-monitoring.py`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
chore/remove-unused-services
No results found.
Labels
Clear labels   
area/automation
Automation, CI, and scheduled workflows

  
area/dedicated
Dedicated Proxmox server and its workloads

  
area/dns
Recursive and authoritative DNS

  
area/docs
Documentation and runbooks

  
area/identity
FreeIPA, LDAP, OIDC, and access

  
area/media
Media management and playback

  
area/monitoring
Metrics, logs, alerts, and probes

  
area/network
Routing, firewalls, VLANs, and ingress

  
area/security
Security controls and access reviews

  
area/storage
Storage, backup, and recovery

  
priority
p1
 Important operational or security work

  
priority
p2
 Normal planned work

  
priority
p3
 Low urgency or future work

  
type/cleanup
Remove obsolete or unused state

  
type/decision
Requires an explicit design decision

  
type/maintenance
Operational maintenance work

  
type/migration
Migration with a staged cutover

  
type/security
Security review or hardening

No labels
area/automation
area/dedicated
area/dns
area/docs
area/identity
area/media
area/monitoring
area/network
area/security
area/storage
priority
p1
priority
p2
priority
p3
type/cleanup
type/decision
type/maintenance
type/migration
type/security
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
stanta/homelab#18
No description provided.