Incidents and postmortems

An incident in Orbit Pulse is the record of one outage or degradation, from the first alert to the postmortem. This page is for project members who respond to problems and for the people who review them afterwards.

Members and project administrators can declare and change incidents. Every project role can read them. Only administrators delete one.

Declare an incident

You can declare an incident by hand, or from something Pulse already shows:

a monitor, or one alerting group of a monitor;
an error group;
a deployment;
a security signal.

Use Declare incident on that item. The incident links to it, and the services the item concerns are added to the incident.

Field	What it holds
Number	A number per project
Title	Up to 200 characters
Severity	SEV-1 (most severe) to SEV-5. The default is SEV-3.
Status	`active`, `stable` or `resolved`. Reopening an incident clears its resolved time.
Commander	A project member who can edit. By default the person who declares it. They get an Orbit notification when someone else makes them commander.
Summary	Up to 10,000 characters
Services	Up to 20 affected services
Postmortem	A link to the postmortem page

A project can hold at most 200 open incidents. Find them in Incidents, where open incidents come first, ordered by severity and then by time.

Follow the timeline

The timeline puts what people did next to what the telemetry shows. People’s entries are status, severity, commander, title, summary and services changes, notes, links and follow-up tasks. Pulse adds, from one hour before the incident until it resolved:

state changes of the linked monitors, with the group and values;
deployments of the affected services;
error groups of those services that were first seen in that time.

Add a note at any time with up to 10,000 characters.

Link work and create follow-ups

Link what belongs to the incident: monitors (optionally one group), error groups, deployments and security signals. A link can be removed again. An incident can have up to 100 links.
Link an existing Orbit work item by its key. Orbit links it as you, with your current access to the work item. It also adds a comment that points at the incident.
Create a follow-up task from the incident. Orbit creates a work item of type Task as you, with the project’s own schemes, labelled pulse and incident, and with a link back. An incident can have up to 50 follow-up tasks.

Both appear in the timeline a few seconds later, once Pulse has Orbit’s answer. If Orbit cannot do it, for example because you lack permission, the failure appears in the timeline.

Write the postmortem in Folio

Use Write in Folio on the incident. Orbit Folio opens its postmortem template for that incident and fills it from the incident:

the incident link, severity, status, commander, times, duration and services;
a timeline table in UTC, built from the incident timeline (up to 200 rows);
the linked monitors as chips with live state, and links to the errors and deployments in Pulse;
sections for root cause, what went well and what went wrong.

When you publish the page, Folio records it as the incident’s postmortem, unless the incident already has one. You can also paste the address of any other page as the postmortem link, and remove it again.

Action items from the page become Orbit tasks that are also linked to the incident. See Link pages to work, code and operations.

Ask Lumen what broke

In an incident, Orbit Lumen can answer “What broke?”. It reads the incident, its monitors, the deployments and their commits, related Orbit work and the Folio runbook. The answer cites each source. See How Lumen answers and what it can see.