Skip to content
SDK docs menu

Classification

How the gateway decides who is asking (five checks, cheapest first, pure compute) and the guardrail that keeps crawlers on canonical HTML.

Last updated

On this page
  1. The five checks, cheapest first
  2. From detection to serving path
  3. The Googlebot guardrail
  4. Confidence semantics
  5. Fixture-driven testing
  6. Verification (Web Bot Auth)

The five checks, cheapest first#

Detection is pure compute: no network calls, no crypto, just a handful of header string scans well under 1ms. Every check that fires is recorded in signals; the highest-precedence one classifies.

  1. Accept: text/markdown (explicit in the list, q-values honored, q=0 excluded) → agent, acceptsMarkdown: true, high confidence. Claude Code and OpenCode send this. Platform attribution comes from Signature-Agent or UA if present, else unknown.
  2. Web Bot Auth headers: Signature-Agent (e.g. "https://chatgpt.com" → chatgpt) → agent, high confidence; a bare Signature + Signature-Input pair → agent/unknown, medium confidence. Parsed only by detect(); verified stays false here because cryptographic verification is the separate async verifyWebBotAuth() step (see below).
  3. Protocol route on the URL pathname: /.well-known/ucp, /.well-known/acp, /mcp, /acp/* (each also matching subpaths) → protocol.
  4. UA heuristics: a *fallback only*, capped at medium confidence. Known agent UAs: claude-code, opencode, ChatGPT-User, OAI-SearchBot, GPTBot, PerplexityBot, Perplexity-User, Gemini/Google-Extended, anthropic-ai/claude-web (plus ClaudeBot/Claude-User), meta-externalagent/meta-externalfetcher (Meta), Amazonbot, DuckAssistBot and MistralAI-User. Crawler UAs: Googlebot → crawler/googlebot, bingbot and adidxbot → crawler/bingbot, MicrosoftPreview → crawler/microsoft-preview, Applebot → crawler/applebot. Applebot-Extended is a robots.txt token that never crawls, so it has no pattern.
  5. Default → human. High confidence when a browserish Accept (text/html) or UA (Mozilla/) is present; low otherwise (empty or garbage headers).

From detection to serving path#

The gateway maps the detection onto a serving path (a Response means markdown was served; null means your HTML pipeline runs as if the gateway weren’t there):

DetectionPathBehavior
Agent with Accept: text/markdown or an identified platformmarkdownRender from sources; Response returned
Agent (signed/identified) on a protocol route, no markdown AcceptprotocolRouted to the protocols hook, since signed protocol clients must reach the protocol handler
Agent, unidentified, no markdown Accept (e.g. bare Signature pair)htmlnull, because the gateway never guesses a format nobody asked for
Humanhtmlnull; your HTML pipeline runs as if we weren’t there
Crawler (Googlebot, bingbot, Applebot)htmlnull, always; see the guardrail below
Protocol routeprotocolconfig.protocols hook when wired; null otherwise; the event records the demand either way

The Googlebot guardrail#

Known search crawlers receive canonical HTML. The gateway renders the same source offer in markdown for matching agent requests; a user-agent claim does not authorize access to private terms.

Confidence semantics#

DetectionResult
const result = detect({ headers: req.headers, url: req.url, method: req.method })
// {
//   kind: 'agent' | 'human' | 'crawler' | 'protocol',
//   platform: 'claude-code' | 'opencode' | 'claude' | 'chatgpt' | 'gemini'
//           | 'perplexity' | 'meta' | 'amazon' | 'duckduckgo' | 'mistral'
//           | 'googlebot' | 'bingbot' | 'microsoft-preview' | 'applebot'
//           | 'unknown' | null,
//   verified: boolean,          // ALWAYS false from detect(); verification is the
//                               // separate async verifyWebBotAuth() step
//   acceptsMarkdown: boolean,   // Accept explicitly lists text/markdown with q > 0
//   signals: string[],          // ordered checks that fired, e.g. ['accept:text/markdown', 'ua:claude-code']
//   confidence: 'high' | 'medium' | 'low',
// }
  • high: an explicit negotiation signal (Accept: text/markdown, or an identified Signature-Agent). Also the browserish-human default.
  • medium: UA heuristics and bare signature pairs. These identify a claimed requester, not verified authority.
  • low: the default human classification with empty or garbage headers.
  • Authorize private actions through the appropriate verified credentials and scopes. Web Bot Auth verification is separate from a parsed platform claim.

Fixture-driven testing#

Detection rules are tested against versioned header fixtures. Add a regression fixture when introducing a new rule, and keep verified identity separate from heuristic classification.

Verification (Web Bot Auth)#

Verification is a separate async step from detection: detect() stays synchronous, parse-only, and its verified is always false. verifyWebBotAuth() cryptographically verifies a Web Bot Auth signature chain (RFC 9421 Ed25519 message signatures) against a caller-injected key registry. It is pure crypto over ambient WebCrypto with zero network, and it never throws: every malformed or adversarial header resolves to { verified: false, reason }.

lib/gateway-config.ts
import type { AgentKeyRegistry, GatewayConfig } from '@rebilder/gateway'

const registry: AgentKeyRegistry = { /* operator-populated: your trusted platform keys */ }

const config: GatewayConfig = {
  storeId: 'store_123',
  sources,
  verification: { keys: registry },
}