Evaluate a Plugin: Fixed Suite, PAT Export, Deterministic Metrics

How to evaluate a plugin's skills before you ship a change: run the same prompts through a real client under the old and the new version, then compare the two versions by the facts the gateway recorded.

Last updated: October 30, 2026

On this page

TL;DR: Every conversation that invokes a skill is credited to the marketplace version that served it. So an evaluation is three steps: run a fixed set of prompts under the current version, ship the change so the version hash moves, run the same prompts again, and read Versions → Evaluation. Every figure is a count, a sum or a fixed rule over the gateway's own record of each conversation. No model scores anything, so the same conversations always give the same numbers. Quality review is a person reading the transcripts.

Why this shape

Fixed inputs. Use the same prompts each time, one per skill, with the same constraints in the prompt (scope to the test records, never write, never ask). A change in the figures is then a change in the skills, not in the questions.
A real client. Run the prompts in Claude Code through the Astound bridge, so every turn passes the gateway, governance, cost accounting and the audit record exactly as a user's would. The evaluation measures what users get.
Deterministic figures. Turns, tokens, cost, latency and tool calls come from the gateway's record of each conversation. Error classes are fixed patterns over tool results, applied once by the conversation rollup and stored on the run. Nothing is inferred.
Two histories. The console keeps every version of every marketplace forever, with the conversations credited to it. Your exported transcripts are the private record of what the assistant actually said.

The loop

run the prompt set under the current version
  → edit skills → ship (new image → marketplace hash moves → bridge syncs)
  → run the same prompt set again
  → Versions → Evaluation: version = new hash, against = old hash
  → Versions → Compare for what changed in the bytes
  → open any skill or conversation that moved; export what you want to keep

The first run on a plugin is its baseline. Every later run is read against the version before it.

Step 1: Run the prompt set

Connect a client to the gateway (Connect Claude Code) and, for each skill in the plugin, start a fresh session and invoke the skill by its slash command with its fixed prompt. Let each run finish. Use a test account with the same entitlement as a real user, so the plugin's marketplace is the one being served.

Expected result: within a minute of each run, the conversation is on /admin/analysis/skills/<plugin:skill> with the version hash in its Skills column. Create a Conversation and See It Land has the checks when it is not.

A conversation is credited to the version being served when it first invoked one of the marketplace's skills, and stays there whatever is deployed later. A run made before the deploy lands under the old hash. That is what makes the comparison fair.

Step 2: Ship the change

Change the skills, then deploy. On the next boot or Sync inventory now, the marketplace is hashed again and the new version opens on Versions → History, with the skills added, removed and changed listed under what changed. The bridge syncs the new bundle to each device; Distribution shows which devices have received it.

Then run the same prompt set again, exactly as in Step 1.

Step 3: Read the Evaluation tab

Open /admin/analysis/versions/<marketplace_id>?tab=evaluation. Pick the Version to read (the newest version with conversations by default) and the version to read it Against (by default, the previous version that has conversations). Pick a window that covers both runs.

Every figure on the tab is an aggregate of the analysis-plugin-eval rows, one row per conversation that invoked one of the marketplace's skills, so the page and the export always agree. A run the rollup has not reached yet reads as zero calls until it does.

The headline shows the selected version's conversations and eight figures, each with its change against the baseline, toned good or bad:

Figure	Meaning
Success	Share of runs that succeeded: completed, made at least one call to the plugin's own connector, had no schema errors, wrote nothing, and had its tools available.
Completed	Share of runs that rendered a widget or ended with an answer of at least 200 characters.
Cost per run	Turn spend over runs. Side calls such as titles and summaries are excluded.
Output tokens per run, Cached context per run	Output and cache-read tokens over runs.
Turns per run	Gateway turns over runs.
Connector calls per run	Calls to the plugin's own MCP servers over runs.
Failed calls	Tool calls whose result fell in an error class, with the split by class beneath: schema, access, upstream, bad arguments, timeouts.

Every version with conversations, oldest first, follows with the same columns, each row compared with the one before it. Then each skill under the selected version, compared with the same skill under the baseline: runs, success, cost per run, output tokens, turns, connector calls and failed calls, with a link to the skill's latest conversation.

The error classes are fixed patterns over the tool result, the same in the console and the export:

Class	Rule
Schema error	`INVALID_FIELD`, `No such column`, `No such relation`, `INVALID_TYPE`, `is not supported`, `MALFORMED_QUERY` or `ValidationError`.
Access error	`INSUFFICIENT_ACCESS`, `Permission '…' denied` or `PERMISSION_DENIED`.
Upstream error	A 5xx `status_code` or `UNKNOWN_EXCEPTION` from the connector's backend.
Bad arguments	`Invalid JSON payload`, `Invalid value at '…'`, or a name that does not match its pattern.
Timeout	`timed out`.
Write	A call to a create, update, delete, upsert, send or insert tool, or with a POST, PATCH, PUT or DELETE method.
Repeated call	A call re-issued with identical input in the same conversation.
Tools unavailable	The final answer says a tool or connector was not available.
Placeholder mention	`MOCK`, `placeholder` or `sample data` in the final answer.

Read the skill table first. A skill whose success fell or whose cost per run rose is the one to open.

Step 3b: Compare the bytes

?tab=compare puts the two versions side by side and lists every skill either version carried with its change (added, removed, changed or unchanged, from the two manifests' per-skill digests) and its invocations, users, requests, failures, cost and p95 under each. An unchanged skill whose figures moved tells you the cause is elsewhere: a model change, a connector, or a different prompt. History gives the figures per version as a trend. Figures overlap across skills in one conversation and must never be summed.

Step 4: Open what moved

The skill. /admin/analysis/skills/<plugin:skill> splits the skill's invocations by client, model or version, and lists every conversation that invoked it: person, client and model, turns, tool calls, artifacts, errors, tokens, cost, p95 and length. A skill that keeps a workflow log also shows its runs by kit version: how many reached the end state, median and p90 time, agent time against human wait, prompts, questions and cost per run.
The conversation. /admin/analysis/conversations/<context_id> shows every plane: per-turn tokens, cost and latency, the turn ledger with model, status and finish reason, every tool call with its execution and result, every governance decision, every safety finding, and the skills invoked with the version served. This is where you read what the assistant actually did.
The revision. From a marketplace's Distribution view, a revision link opens /admin/analysis/revisions/<id>, which lists every file of the published skill with its size, digest and a text preview. Use it to confirm the bytes that were served are the bytes you changed.

Step 5: Export and keep the evidence

A personal access token (sp-live-…) is accepted on GET requests to /admin/export/… only. It resolves to its owner and sees exactly what that person's console session would. Issue one under Account → Devices → Personal access tokens, or use the bridge's own token.

Request	Returns
`GET /admin/export/transcripts?session_id=<client session id>`	JSON Lines, one `ConversationBundle` per conversation in that session.
`GET /admin/export/transcripts/{context_id}?format=json\|markdown`	One conversation.
`GET /admin/export/analysis-skills?from=…&to=…&format=json`	One row per skill: invocations, people, requests, tokens, cost, tool failures, p95.
`GET /admin/export/analysis-skill-conversations?skill=<plugin:skill>&from=…&to=…`	One row per conversation that invoked the skill.
`GET /admin/export/analysis-skill-runs?skill=<plugin:skill>&from=…&to=…`	One row per workflow run of the skill.
`GET /admin/export/analysis-conversations?from=…&to=…&format=json`	One row per conversation in the window.
`GET /admin/export/analysis-plugin-eval?marketplace=<id>&days=30&format=json`	The Evaluation tab's rows: one per conversation that invoked the marketplace's skills, with version, tokens, cost, turns, connector calls, each error class, writes, repeated calls, `completed` and `success`.
`GET /admin/export/analysis-plugin-eval-tools?marketplace=<id>&days=30&format=json`	One row per version, skill and tool: calls, failed calls by class and repeated calls, by the same rules. It names the tool that is failing, not just the skill.

Every dataset also takes format=csv|json|jsonl|markdown, and /preview returns the row count without the body.

curl -H "Authorization: Bearer $PAT" \
  "https://<gateway>/admin/export/transcripts?session_id=<id>" -o run.jsonl

A bundle carries facts (tokens, cost, turns, latency, tool calls, governance decisions), requests (per turn), tool_calls and tool_ledger, skills (each invoked skill with its plugin_id, marketplace_id and marketplace_hash), decisions, messages and transcript. The hash in skills is the key that ties an exported conversation to a row on the Evaluation tab. Export on that tab downloads analysis-plugin-eval for the same window.

Keep exports off the repository. Transcripts describe real users' work, even when the user is a test account.

Reading the result

The figures tell you whether the change did more work, failed more, cost more or took longer. They do not tell you whether the answer was right. For that, open the transcripts of the after run and read them against what the prompt asked for. A change ships when success and cost did not regress against the baseline and a person is satisfied with the transcripts.