Architecture: Host / Runner / TestSuite

How the conformance harness is factored so any host — a web chat client today, a desktop app like VSCode or Goose tomorrow — can run the same suite. All the platform-specific code lives behind one Host interface.

Three objects

                 in the iframe            external process (a "Host")
              ┌────────────────────┐      ┌──────────────────────────────┐
   asserts →  │      TestSuite      │      │           Runner             │
              │  owns test defs +   │      │  generic dispatch loop:      │
              │  MCP-app comms      │      │  poll → dispatch → resolve   │
              │                     │      │            │                 │
              │  window             │◀─────┤  SuiteBridge (frame.evaluate │
              │  .__mcpConformance  │─────▶│    for a browser host)       │
              │  listTests/start/   │ poll │            │                 │
              │  poll/resolve       │      │            ▼                 │
              └─────────┬───────────┘      │   Host capability methods    │
                        │                  │  clickTrigger / confirmDialog│
              ext-apps App               │  conversationContains /      │
              (PostMessage bridge)         │  toggleTheme / …             │
                        │                  └──────────────┬───────────────┘
                        ▼                                 │ real clicks, DOM,
              ┌────────────────────┐                      │ conversation API
              │  the chat host      │◀─────────────────────┘
              │  (ChatGPT / Claude  │
              │   / a desktop app)  │
              └─────────┬───────────┘
                        │ MCP
                        ▼
              ┌────────────────────┐
              │   MCP server        │
              └────────────────────┘

The typed capability protocol

The protocol is deliberately primitive — one variant per thing a host can physically do. The test carries the meaning (e.g. "the tool must stay hidden" is a test that negates a conversationContains result, not a dedicated request kind).

// shared/protocol.ts — the one contract both sides import as source
export type CapabilityRequest =
  | { kind: "clickTrigger"; commitDraftedMessage?: boolean }   // real cross-origin click (user activation)
  | { kind: "confirmDialog"; dialog: "open-link" | "download" | "sampling" }
  | { kind: "conversationContains"; marker: string; timeoutMs: number }
  | { kind: "toggleTheme"; to: "light" | "dark" }
  | { kind: "readModelToolList" }        // optional desktop-host affordance
  | { kind: "resetIsolation" };          // suite emits this before each manual test

export interface CapabilityResult {
  ok: boolean; value?: unknown; error?: string;
  unsupported?: boolean;                 // host lacks the capability → test skips / falls back
}

Pull model: the suite pulls, the Runner polls

A sandboxed, nested, cross-origin iframe can't reliably push a message out (nested cross-origin postMessage drops), but the Runner can always reach in via frame.evaluate. So the direction is fixed: a test awaits t.host(req), which parks req in a single pending slot; the Runner poll()s that slot, services the request against the Host, and calls resolve(result) to unblock the test.

Optional capabilities & fallback

Only setup() and teardown() are mandatory on a Host; every capability method is optional. An absent method resolves to { unsupported: true }. In the suite:

• t.host(req) — auto-skips the test if the capability is unsupported ("I need this or I can't run here").
• t.hostOptional(req) — returns the raw result so the test can fall back to another path.

// visibility/app-tool-hidden — capability-or-fallback
const direct = await t.hostOptional({ kind: "readModelToolList" });
if (!direct.unsupported) {               // a desktop host that can introspect the model's tools
  t.assert(!(direct.value as string[]).includes("conformance_probe"), "hidden tool leaked");
  return;
}
// browser hosts don't expose that — fall back to asking the agent and scanning the conversation
t.bindTrigger(() => t.app.sendMessage({ role: "user", content: [{ type: "text", text: ASK }] }));
await t.host({ kind: "clickTrigger", commitDraftedMessage: true });
const r = await t.host({ kind: "conversationContains", marker: "conformance_probe", timeoutMs: 45_000 });
t.assert(!r.ok, "hidden tool name surfaced in the conversation");

Pluggable hosts

The three chat products differ in behavior — how you enter a prompt, dismiss a modal, verify a conversation turn, commit a drafted message — and in which capabilities they support. So they are subclasses of a shared BrowserHost, not config rows:

VSCodeHost / GooseHost would be peer Host implementations — not BrowserHost subclasses — providing their own setup()/SuiteBridge transport and whichever capabilities they can (a desktop host may implement readModelToolList that browser hosts can't). Anything a host can't do is simply unsupported, and those tests skip. Where the browser driver clicks real product UI, it necessarily overfits per-host DOM — see How it works.