How to read this
The words
| Word | Means |
|---|---|
| Server | one MCP address |
| Check | one run of the full set of questions, six runs each, on one model |
| Question | one request text, written by a model and spot-checked |
| Group | questions that share one right tool |
| Run | one answer to one question |
| Tool | one entry in the server's list |
| State | OK, Watch or Fail; Unsure under three runs |
One cell per run
| Cell | Result | Means |
|---|---|---|
| Right first | the first tool the agent called was a right one | |
| Wrong turn | it called something else first, then reached a right tool | |
| Never right | it finished without calling a right tool; answering with no tool counts here | |
| Failed | the model call itself failed; not counted either way |
States
| State | A question | A server |
|---|---|---|
| OK | right tool first in every run | right tool first in at least 9 of 10 runs, no question Fail |
| Watch | right tool first in some runs | 7 to 9 of 10, or a question Watch |
| Fail | never right first, or a run never reached a right tool | under 7 of 10, or any question Fail |
A group's state is its worst question's. A state is a fact about one check, not a grade of the server.
The numbers
Each question is asked six times. Six is a fixed budget, not a proof: 6 of 6 is consistent with a tool that is right most of the time, and one odd run can be chance. Read the counts as what happened on those runs.
Median s is the median time of a run, in seconds. Calls is how many tools a run called before it answered.
The harness
The agent sees the server's tools and nothing else, and gets at most 5 turns per run. Only read-only tools are called; anything that could change data is listed on the server page as a tool not called, and no question counts on it. Models used so far: Haiku 4.5. The server's replies are not republished: a run shows the arguments the agent sent, the size of what came back, and the agent's answer with any code removed.
What this cannot tell you
The same model wrote the questions and which tools count as right. That can make OK too generous; it cannot invent a miss. The questions are spot-checked before a check is published, obviously wrong ones fixed or dropped, but that is a filter, not an independent review.
Each check is one model, one date, one server version. A different model, or the same server a month later, can behave differently; the date and version are on every server page.