Administrator console and quality checks

This page is for Orbit administrators. The Lumen panel has an Admin tab that only administrators see. There you control Lumen, watch how well it answers and check that a model change did not make it worse.

Model, provider, key and budget settings, per-product switches and conversation retention are on Models, providers and cost. This page covers the views for usage, quality and changes.

View in the Admin tab	What it is for	Status
Answer quality	How often answers cite sources, how often Lumen says “not found”, and how people rate answers	Built
Quality checks	Your own reference questions, run on a model and compared with the last run	Built
Changes	The log of every card that was proposed, run or dismissed	Built
Usage	Calls, tokens and cost by model, function, person and agent	In development
Explain access	Why a person has no access	In development

Read answer quality

Open Answer quality to see how Lumen is doing. The numbers cover:

the answers given, and the share that carry sources;
“not found” answers, which are answers without links that say nothing was found;
empty answers and refusals;
ratings by model and by reason, and the latest answers rated “not helpful”, without names;
the same figures day by day, with the time and cost of one answer.

People rate an answer with a thumbs up or down, a reason and a comment. The report shows no names, and it stores no question or answer text. Quality-check runs are not counted.

Run a quality check

A quality check is a list of questions you wrote, each with the words a good answer should contain. Run it after you change anything that may affect answers, such as the model.

Open Quality checks and add your reference questions. You can store up to 50. Give each the words the answer should name.
Run the questions on the model Lumen uses now. On a customer-run installation Planned, the console can also run them on another provider and model for that run only.
Read the comparison. Lumen compares each run with the one before it and shows what got worse and what got better. It keeps the last 30 runs.

Lumen asks the questions as you, without saving them in any conversation history.

Review what Lumen proposed

Changes lists every card. It shows what was proposed, in which answer, and whether it was run, dismissed or failed. It also shows whether the person edited it, with a link to the result. Authors see their own entries and administrators see all of them. See Actions with confirmation cards.

See usage In development

Usage shows calls, tokens and spend over 7, 30 or 90 days, a chart by day, and tables by model, function, user and agent. For each agent it also shows event runs by outcome, such as done, failed or limited. It has passed tests on recorded answers. A check against live usage data is still ahead.

Ask why someone has no access In development

In Explain access you ask why a person cannot see something. Lumen reads the person’s Orbit account (role and whether it is disabled, without e-mail) and the members and roles of the project. It also reads the security of the work item: classification, allowed users and archive. It then explains the cause and the steps to fix it.

These tools are for administrators only. A member is not offered them, and cannot run them. A live run and an evaluation question are still ahead.

Explain a setting In development

Lumen can explain how a project’s workflow, fields and screens work now. It reads the default workflow and the workflow for each work item type. It also reads the statuses, the transitions with their required fields, the status catalog, and the create and edit screen of a work item type. It can compare a workflow draft with the published one. A pipeline is read from the CI file.

A preview of what a setting will change before you publish is still ahead.