Context7 vs Web Search — Token & Speed Test

Method

Four questions a developer would plausibly ask mid-task were run through both pipelines, back to back, in one Claude Code session:

• Q1 — How do I use load functions and form actions in SvelteKit?
• Q2 — How do I set and read cookies in hooks and load functions in SvelteKit?
• Q3 — How do I handle errors and retries in Effect?
• Q4 — How do I define services and layers with dependency injection in Effect?

Each Context7 run = one resolve-library-id call + one query-docs call (the prescribed per-question flow). Each web run = one WebSearch + one WebFetch of the best result (the fetch is summarized by a small model before it reaches the agent, mirroring how Claude Code's WebFetch actually works). Raw tool outputs were saved verbatim to markdown files and tokenized with tiktoken (o200k_base). Wall-clock timestamps were logged around every call.

Token counts

Tokens per tool output, o200k_base encoding. Step columns are the two tool calls in each pipeline.

Speed

Wall-clock between bracketing timestamps; every interval includes ~2–4s of agent inference overhead, so absolute values are inflated but the comparison is fair. *Q4 web fetch is additionally inflated by extra agent work at the end of the run.

Context7 won on latency in every single pairing. Individual Context7 calls landed around 7–12s; web search and the WebFetch summarization step (which runs an extra model pass over the page) each ran roughly 13–16s. A two-call Context7 question costs about 20–25s end to end versus 30–35s for search + fetch.

Content quality

The token counts only tell half the story — what you get per token differs a lot:

• Context7 returns verbatim, current code snippets straight from indexed docs, with source URLs. Q4 (Effect services/layers) pulled the curated effect-solutions patterns including full working Context.Service + Layer.effect implementations — high signal for the agent to write code from.
• Web returns a search summary plus a small-model digest of one page. It is cheaper and reads more like an explanation, but it is lossy (the model paraphrases code), covers only the one page you fetched, and the search step spends ~600–750 tokens largely on link metadata.
• Noise exists on both sides. Context7's Q1 answer mixed in SvelteKit's newer remote-functions API alongside classic form actions, and one Effect snippet had a mangled import line; the web digest for Q3 was accurate but shallower, deferring to "see the official docs."

Context7 tokens spent on code

Majority — snippets dominate query-docs output

Web tokens spent on overhead

~58% of web total is the search step (links + summary)

Freshness

Both current; Context7 pins to indexed doc versions, web reflects live pages

Caveats

• n = 4, single run. This is a vibe check, not a benchmark. Variance between runs (network, search ranking, doc index) is untested.
• Tokenizer is a proxy. tiktoken's o200k_base (GPT-4o encoding) was used because Anthropic doesn't ship a local tokenizer; Claude's tokenizer typically counts somewhat higher, but the ratios between pipelines hold.
• Timing includes agent overhead. Each interval contains one model inference gap (~2–4s). Relative comparisons are fair; absolute latencies are upper bounds.
• Tool-output tokens only. Tool schemas/definitions sitting in the system prompt (both MCP tools and WebSearch/WebFetch) aren't counted, nor are the small request payloads.
• Web workflow shape matters. One search + one fetch is a lean baseline; real sessions often fetch 2–3 pages, which would widen the web token count quickly (~300–600 tokens per extra fetch).

Takeaways

Pinned-ID time estimated as the measured query-docs intervals alone.

Raw materials live in ~/Developer/sandbox/deej/context7-token-test/: per-query markdown transcripts (context7/, websearch/), timing.log, analyze.py, and results.json.