← Results

How the conformance suite works

Generated 2026-07-22 17:41

The pipeline

  1. The MCP server exposes a ui:// runner resource.
  2. The host renders it in a sandboxed iframe — the TestSuite, which owns the test definitions and the ext-apps App (MCP-app) communication.
  3. A test asserts the host's behavior against the spec; when it needs the host to do something it can't do from inside the iframe, it emits a typed CapabilityRequest and awaits the result.
  4. The suite installs one control seam at window.__mcpConformance (listTests / start / poll / resolve). The Runner polls it (via frame.evaluate), dispatches each request to the active Host, and resolves it back — the suite pulls, the Runner polls.
  5. The Host is the only platform-specific piece (a Playwright BrowserHost today; a desktop host later). It automates the real host — clicks, dialogs, conversation checks — and returns each result.
  6. report.ts renders the results matrix (Results); see also the architecture.

The durable seam is the window.__mcpConformance protocol. The fragile part is the per-host DOM the driver must click — see the last section.

Test buckets

Automatic 22

Asserted directly from inside the iframe — read a host value, proxy a tool call, trigger a CSP violation. No human, runs headless.

lifecycle/initialize-capabilitiesMUST2026-01-26 · L624
lifecycle/tool-inputMUST2026-01-26 · L1106
lifecycle/tool-input-partial-stopMUST2026-01-26 · L1136
lifecycle/tool-resultMUST2026-01-26 · L1155
tools/proxy-callMUST2026-01-26 · L487
visibility/app-tool-call-guardMUST2026-01-26 · L401
display/return-resulting-modeMUST2026-01-26 · L787
dimensions/listen-size-changedMUST2026-01-26 · L718
security/sandbox-distinct-originMUST2026-01-26 · L474
security/sandbox-permissionsMUST2026-01-26 · L475
security/csp-construct-from-domainsMUST2026-01-26 · L479
security/csp-allow-declaredMUST2026-01-26 · L479
display/no-undeclared-modeMUST NOT2026-01-26 · L786
security/csp-no-looseningMUST NOT2026-01-26 · L286
context/initialize-hostcontextSHOULD2026-01-26 · L533
context/light-darkSHOULD2026-01-26 · L895
display/unavailable-returns-currentSHOULD2026-01-26 · L788
capabilities/server-passthroughSHOULD2026-01-26 · L487
context/theme-variablesMAY2026-01-26 · L793
context/theme-fontsMAY2026-01-26 · L922
capabilities/content-modalitiesMAYdraft · L663
security/sandbox-proxy-requiredMUST2026-01-26 · L472

Auto-detect (await) 2

The app triggers an action, then a host notification or callback settles the test automatically — no verdict to give.

context/context-changedMAY2026-01-26 · L1229
app-tools/call ask-agentMAYdraft · L1243

Operator / driver verdict 9

The effect happens outside the iframe (a permission dialog, a new tab, a conversation turn); a human or the Playwright driver confirms it from the host surface.

visibility/app-tool-hidden ask-agentMUST NOT2026-01-26 · L400
links/open-externalSHOULD2026-01-26 · L996
messages/add-to-conversationSHOULD2026-01-26 · L1033
model-context/provide-future-turns ask-agentSHOULD2026-01-26 · L1097
sampling/create-messageSHOULDdraft · L534
download-file/confirmSHOULDdraft · L1128
security/iframe-sandboxedMUST2026-01-26 · L1698
security/csp-audit-logSHOULD2026-01-26 · L287
model-context/last-wins ask-agentSHOULD2026-01-26 · L1101

ask-agent is a cross-cutting tag: the app sends a ui/message asking the model to act (call a tool, list tools, recall context), so the result depends on the agent — not just the host bridge.

Where the driver overfits the host

The driver clicks real product UIs, not an API, so it hard-codes host-specific selectors and flows. These are brittle to host redesigns.

Concernchatgptclaudealpic-playground
Launch prompt#prompt-textarea, “run @app” (mention picker, 2× Enter)ProseMirror div[contenteditable], “run app”, 1 Entertextarea[name="message"], Enter
Widget iframeiframe[src*="oaiusercontent"]iframe[src*="claudemcpcontent"]iframe (same-origin)
Dismiss host chrome“Got it” modal“Accept All Cookies” banner—
Permission dialog“Open link” (as <a>), “Download”“Open link” / “Download” (as <button>), “Allow”—
Send a ui/messagesent directlydrafts into composer → “Replace current text?” + button[aria-label="Send message"]—
Verify a conversation turn/backend-api/conversation/<id> APIscrape document.body.innerTextscrape innerText
Reset display modereal-click “Reset to inline” (gesture-gated)programmatic requestDisplayMode works—