An incident is an unplanned interruption or reduction in service quality, detected and tracked through monitoring systems, that requires diagnosis and resolution. In IT operations, incidents range from performance degradation to complete outages; in security, they represent violations or threats to security policies.
Incidents are typically detected through Health Check mechanisms, Uptime monitoring, or alerting systems that observe metrics, logs, or security events. When an alert threshold is crossed—such as error rates spiking, a service becoming unreachable, or unauthorized access—an incident is created in an on-call system (PagerDuty, Opsgenie, etc.) with a severity level assigned.
The incident lifecycle follows a standard pattern:
- Detection: Monitoring rules or anomaly detection trigger an alert
- Triage: On-call responder assesses severity and impact
- Mitigation: Actions taken to restore service (rollback, restart, failover)
- Root-cause analysis: Post-incident review documenting what happened and why
- Closure: Ticket resolved with lessons learned documented
Example: Your monitoring system detects that an application is returning HTTP 500 errors for 10% of requests. An alert fires, an SEV-2 incident is auto-created, and a notification is sent to the on-call engineer, who begins investigating the database connection pool exhaustion.
WarningNot every alert is an incident. Alert fatigue from over-sensitive thresholds can cause real incidents to be missed. Implement proper alert deduplication, severity classification, and escalation logic to ensure only genuine service issues become tracked incidents.
Related concepts: DNS Failover for automatic recovery, MTTR for measuring resolution speed, and Status Page for communicating incidents to users.