Logs: explorer, pipelines and patterns
Orbit Pulse keeps the logs of a project in one explorer. This page is for members who search logs every day and for project administrators who decide what is stored.
If you have not searched logs yet, start with Send your first logs. The search language is in the query syntax reference.
Save a view
Section titled “Save a view”A saved view is a named query with its time range, facets and columns. Everyone in the project can open it.
-
Run a search in Logs. Pick the time range, the facets and the columns you want.
-
Use the Views menu beside the search to save the screen as a view.
-
Later, open the same menu to switch views. A view you changed on screen is marked edited, and the menu offers Update and Save as new view.
| Rule | Value |
|---|---|
| Name | 1 to 100 characters, unique in the project ignoring case |
| Views per project | At most 200 |
| Who saves | Members and project administrators. The author or an administrator deletes. |
| Range | A relative range (15 minutes, 1 hour, 4 hours, 1, 7 or 15 days) or a fixed window of at most 32 days |
| Default view | Each person, viewers included, picks their own. The Logs tab opens with it unless a link says otherwise. |
Saving over a view that someone else changed in the meantime is refused, so nobody overwrites another person’s edit.
Add columns and export
Section titled “Add columns and export”Show as a column on any attribute or resource tag of an open log adds it to the list. Remove it from the column header. With columns shown, the message moves to its own line.
Export CSV downloads the newest 10,000 matching logs. The file has the time (UTC), status, service, host, the columns you show and the message. A cell that would run as a spreadsheet formula starts with a quote, so opening the file is safe.
Read patterns
Section titled “Read patterns”Switch Logs to Patterns to see similar messages grouped together, most frequent first. Pulse groups the newest 50,000 logs that match your query. If more match, the page says so and since when.
A pattern is the first 512 characters of the message with variable parts masked:
| Part | Shown as |
|---|---|
| URLs with a scheme | <url> |
Query strings with a key=value |
?<query> |
| UUIDs | <uuid> |
| ISO, access-log and slashed times | <time> |
| IPv4 addresses, with an optional port | <ip> |
| Email addresses | <email> |
| Hex tokens with a digit and a letter | <hex> |
| Numbers | <num> |
Each pattern shows its count and share, and its errors and warnings. It also shows up to five services, the first and last time, the newest sample and a trend of 24 steps. Show logs opens the logs of one pattern. Exclude hides a noisy pattern while you stay in Patterns.
Behind this is the pattern:<id> field. It works in search, histogram, facets, export, saved views, log monitors (also grouped by pattern) and live tail.
Process logs with pipelines
Section titled “Process logs with pipelines”Project administrators open Pipelines to change logs before they are stored. Everyone in the project can read the rules and try them.
Each log goes through the rules in this order:
- Pipelines, in their order. Each runs on the logs its filter matches.
- Log-based metrics, counted from every processed log, including logs an exclusion filter drops.
- Exclusion filters. The first filter a log matches decides.
- The daily quota, which counts only the logs that are kept.
Changes reach intake within 10 seconds. A rule has a filter in the query syntax (empty matches every log, at most 2,000 characters). A project can have up to 50 rules of each kind, and a pipeline up to 20 processors.
| Processor | What it does |
|---|---|
| Grok parser | Up to 10 rules. The first rule that matches the whole value wins. There are 24 named patterns, such as IP, HTTPDATE, URIPATHPARAM and LOGLEVEL. Captures become attributes, and int or float captures can be searched as numbers. |
| Remapper | Moves or copies the first source that is set (1 to 10 sources) to a column, an attribute or a tag. The status column accepts only known levels. |
Try a draft before you save it. Paste up to 50 lines or JSON logs, or let Pulse use the newest 20 stored logs the filter matches. For each log you see which fields changed. Nothing is stored.
Drop noise with exclusion filters
Section titled “Drop noise with exclusion filters”An exclusion filter drops every matching log, or keeps a share of them (keeping a quarter, for example). The share is deterministic by log id. A preview counts what the filter would have dropped over the last 24 hours.
Dropped logs never reach storage and do not count against the daily quota. Live tail still shows them.
Make metrics from logs
Section titled “Make metrics from logs”A log-based metric is a count, or a distribution of a numeric attribute, grouped by up to five fields. Pulse writes one point per minute and group into Metrics, within the series limit. Open in Metrics charts it. See Metrics, dashboards and SLOs.
See what a rule does
Section titled “See what a rule does”Each rule shows the logs it matched today and in the last 7 days. A processor shows how many logs it changed, an exclusion how many it dropped, and the page totals the processed and excluded logs. Project administrators create, edit, reorder, disable and delete rules.