Monitors and alerts

A monitor evaluates a query every minute and keeps a state for each group it watches: OK, WARN, ALERT or NO_DATA. Project members and administrators create and edit monitors; everyone with a project role can read them. A project can have up to 500 monitors.

Monitor types

Type	Value per group
log_count	Logs matching the query in the window
apm_error_rate	Failed entry spans divided by entry spans, in percent
apm_latency	A percentile of entry-span duration in ms: p50, p75, p90, p95 (default), p99, avg or max
apm_throughput	Entry spans per minute
error_tracking	Error groups that are new (new, the default), regressed (regressed) or either (any) in the window. Groups by service only.
ci_pipeline_failure_rate, ci_pipeline_duration, ci_job_failure_rate, ci_job_duration	Forge pipelines and jobs, which need Forge next to Pulse. A job with allow_failure is not a failure.
metric	A metric query over the window, with the query as its tag filter
slo	Burn rate (the lower of the long and short windows, so it fires only while both burn) or the share of the error budget used

Thresholds and windows

• Group by up to 4 fields (reserved fields or @attribute). Each combination of values is its own group, keyed field:value,field:value.
• Comparator: above (default), above_or_equal, below or below_or_equal.
• Thresholds: an alert threshold is required. A warning threshold must be less severe. A recovery threshold (hysteresis) must lie on the OK side of the alert threshold; without it, a group recovers as soon as it stops alerting.
• Evaluation window: 60 to 86,400 seconds in whole minutes. The default is 300 seconds, so monitors do not react to a single minute.
• Delay: 0 to 900 seconds, to allow for late data.
• NO DATA policy: a group with no data for noDataSeconds (60 to 86,400) follows noData: ok (the default), no_data, alert or keep. The default waiting time is the larger of 600 seconds and twice the window. Count types treat a missing group as 0.
• OK groups with no data for 24 hours are removed. Each evaluation tick runs exactly once, even with several Pulse instances, so a restart never duplicates a transition or a notification.

You can preview an unsaved monitor to see what it would see now, and validate it, which checks it and reports the first problem with the field and, for queries, the position.

Notifications

A notification is sent when a group’s notified state changes: alert, warn, no_data, recovery, and renotify every 5 to 10,080 minutes while it stays in alert.

• Recipients: up to 20 Orbit users who can read the project, notified in Orbit.
• Webhooks: up to 10 per monitor, signed. The signing secret is shown once when the webhook is created and can be rotated. A project can have 50 webhooks, managed by administrators; a test sends a ping.
• Batching: changes for the same recipient within 30 seconds become one Orbit notification or one webhook call.
• Message template (up to 4,000 characters), with variables such as {{monitor.name}}, {{state}}, {{previous_state}}, {{value}}, {{threshold}}, {{group}}, {{group.<field>}}, {{link}}, {{runbook}}, {{time}} and {{project}}. Sections such as {{#is_alert}}...{{/is_alert}} (also is_warning, is_no_data, is_recovery, is_renotify) show text only for that state, and {{^is_alert}}...{{/is_alert}} inverts it.

Mute and downtimes

A muted monitor, or one inside an active downtime, keeps evaluating but sends nothing. When the mute or downtime ends, recipients hear about whatever changed in the meantime. Downtimes cover one monitor or all of them, can be limited to a scope such as service:checkout, and can recur on chosen weekdays with a start time, duration and time zone. A project can have up to 200 current downtimes.

Runbook URL

Set runbookUrl to an https:// or http:// address, or to an Orbit path such as a Folio page (/folio/p/<pageId>). A Folio page that is a monitor’s runbook shows the live state of every monitor that points to it; see Link pages to work, code and operations.

Automatic Orbit bug

With create bug on, a monitor files one open Orbit bug per alerting group, as the monitor’s owner (with the labels pulse and monitor). When the group alerts again, recovers or is still alerting at a reminder, the monitor comments on that bug instead of opening another. Once the bug is done, the next alert files a new one. See Errors and Orbit bugs.

Cut the noise

• Cascade grouping puts cascading failures into one notification.
• The noisy-monitor report lists monitors by transitions, alerts and flaps over 1 to 30 days (default 7); a flap is an alert within 30 minutes of recovering.
• Monitors can be searched by name, query or tag, and filtered by state, type, owner and mute.

Anomaly monitors

Instead of a fixed threshold, an anomaly monitor compares each group with its own baseline. It works for log_count, the apm_* types, metric and the ci_* types.

Setting	Options
Baseline	recent: the 24 windows just before. daily (default): the same time on each of the previous 7 days, with the two neighbouring windows (21 windows). weekly: the same time over the previous 4 weeks (12 windows).
Direction	above, below or both (default)
Window limit	At most 1 hour for recent and daily, 6 hours for weekly
Thresholds	Counted in deviations: alert 3 by default; alert, warning and recovery range from 0.5 to 10

The expected value is the median of the baseline, and the spread is 1.4826 times the median absolute deviation, with a floor so that quiet series do not produce false alerts. A group needs enough baseline windows with data (12 for recent, 9 for daily, 6 for weekly) before it can alert. Until then it stays OK and is marked as learning. A group seen in the baseline but silent now is also scored, so a drop to nothing alerts with below or both. Notifications and bugs state the expected range, for example “expected 10 (0.51-19.49), 15.8 deviations above”, and charts show the expected band.

Status

All monitor types and features on this page are built.