Evaluation
Raybot was evaluated once, on 2026-06-07 at 12:47 PM, by Darbot. It scored 0 percent, with all six conversations marked Fail.
That number is not a measure of the agent. It is a measure of the evaluation runtime.
What the score actually means
Two things were tested, and both failed for reasons unrelated to grounding quality.
| Target | Result |
|---|---|
| Draft agent | Returned the system fallback response instead of a grounded answer |
| Published agent | Returned HTTP 500 |
Meanwhile the Preview pane, exercising the same agent against the same knowledge source in the
same session, grounded correctly and returned four docs.ray.io citations.
An agent that grounds correctly in Preview and returns a fallback under Evaluate has an evaluation harness problem, not a knowledge problem. The 0 percent is recorded here because suppressing it would be dishonest, and qualified here because reporting it bare would be misleading. The full argument, including the evidence that would overturn it, is in Results.
Pages in this section
| Page | What it covers |
|---|---|
| Test set | The CSV contract, all six question and answer pairs verbatim, why those six, and guidance for authoring replacements |
| Results | The run record, the Preview contrast, falsification criteria for the harness conclusion, and diagnostics |
Evaluate only unlocks after first save
Like Publish and Share, the Evaluate surface is inert until the agent record exists in Dataverse. The agent ID is minted by the first save, and every downstream surface keys off it. See Phase 1.
Endpoints
The evaluate domain accounts for 7 of the 57 mapped endpoints. The v2 evaluation query path
.../makerevaluations/queries/generate?ApplyV2Migration=true was one of four observations
corroborated exactly against the authoritative corpus. See the
endpoint map for the full inventory.
Reproducing this
- Save the agent at least once so Evaluate unlocks.
- Import
raybot-evalset.csvfrom the source repository. - Run against Draft, then against Published.
- Capture both result sets and the Preview transcript for the same questions.
Step 4 is the important one. Without the Preview transcript there is no evidence distinguishing a bad agent from a bad harness.