Pinned Context7 vs Web Search — Round 2
Six day-to-day questions across the pinned library list, each answered by one pinned query-docs call vs one generic web search.
With pinned IDs, Context7 is now a single call that beats web search on speed in all six match-ups (~12.2s vs ~17.8s avg, −31%). It still costs more tokens on paper (6,485 vs 4,432, +46%) — but one Expo outlier with duplicated snippets accounts for most of the gap. Exclude it and the two are near token parity (841 vs 747 avg), with Context7 returning runnable code where web search returns prose summaries.
Method
Round 1 measured the old two-step Context7 flow (resolve + query) against search-plus-fetch. This round tests the workflows as they exist after the pin-first policy: each pipeline gets exactly one tool call per question.
- Context7: one
query-docscall straight to the pinned library ID — noresolve-library-id. - Web: one
WebSearchcall (link results + synthesized summary) — no page fetch.
Six questions, one per pinned source: React (useOptimistic), Vite (env vars), Convex (crons), Expo (push notifications), PlanetScale (safe migrations), and Claude Code (custom slash commands). Raw outputs saved verbatim and tokenized with tiktoken (o200k_base); wall-clock logged around every call (each interval includes ~2–3s of agent inference overhead, so absolutes are upper bounds but the comparison is fair).
Token counts
| Query | Context7 (pinned) | Web search | Δ (C7 − Web) | Cheaper |
|---|---|---|---|---|
| React — useOptimistic | 1,312 | 855 | +457 | Web |
| Vite — env vars | 893 | 698 | +195 | Web |
| Convex — crons | 862 | 616 | +246 | Web |
| Expo — push notifications | 2,279 | 699 | +1,580 | Web (outlier) |
| PlanetScale — safe migrations | 453 | 713 | −260 | Context7 |
| Claude Code — slash commands | 686 | 851 | −165 | Context7 |
| Total | 6,485 | 4,432 | +2,053 |
Tokens per tool output, o200k_base encoding. Without the Expo outlier: 4,206 vs 3,733 (avg 841 vs 747 — near parity).
Context7's Expo answer returned the same ~120-line push-notification example twice — once from the SDK 53 llms.txt and once from SDK 55 — burning ~1,100 redundant tokens. That's a dedupe gap in the source index, not a flaw in the question. Worth knowing: heavily versioned llms.txt sources can double-serve near-identical snippets.
Speed
Context7 won every pairing — fastest pinned call 10.5s vs fastest search 16.6s; the slowest Context7 call (14.6s) still beat the fastest web search. Removing the resolve step cut Context7's per-question latency roughly in half versus Round 1 (22.9s → 12.2s). Each measured interval includes one agent inference gap, so true network latencies are a few seconds lower across the board.
What you get per token
The shape of the output differs more than the counts suggest:
- Context7 returned verbatim, runnable code in 5 of 6 answers — complete
crons.tsfiles, workinguseOptimisticcomponents, full Vite configs — each with a source URL. The PlanetScale answer was concise prose (453 tokens, the cheapest result in the whole test) but accurately covered ghost tables, DDL rejection, and the deploy-request flow. - Web search returned prose summaries with at most a stray inline snippet, plus ~10 link objects per query (roughly 200–300 tokens of each answer is link JSON). Good orientation, but an agent writing code still needs a follow-up fetch — which is exactly the +300–600 tokens and +15s that Round 1 measured.
- Both were accurate. No hallucinated APIs on either side; web summaries occasionally hedged ("refer to the official docs"), Context7 occasionally over-delivered (Expo).
vs Round 1
| Workflow | Tokens / query | Time / query | Calls |
|---|---|---|---|
| R1: Context7 (resolve + query) | ~1,594 | ~22.9s | 2 |
| R1: Web search + page fetch | ~1,142 | ~34.8s | 2 |
| R2: Context7 pinned | ~1,081 (841 excl. outlier) | ~12.2s | 1 |
| R2: Web search only | ~739 | ~17.8s | 1 |
Round 1 used 4 SvelteKit/Effect questions; Round 2 used 6 questions across different pinned sources — directionally comparable, not identical workloads.
The pin-first policy delivered what it promised: Context7's per-question cost dropped ~32% in tokens and ~47% in time versus Round 1. Web-search-only is cheaper than search+fetch, but it's also a weaker artifact — a summary rather than code.
Takeaways
The pinned setup is the right default. For library questions it's ~31% faster than even a bare web search, returns code instead of prose, and runs at rough token parity once you discount one source-side dedupe glitch. Web search remains the right tool for everything unpinned — its summaries are genuinely cheap and accurate for orientation-level questions.
- Watch llms.txt-sourced pins for duplication. The Expo repo serves multiple SDK versions of the same example; if that repeats, consider re-pinning Expo to a docs-site source.
- Prose-heavy questions are Context7's cheapest case — the PlanetScale conceptual answer cost 36% fewer tokens than the equivalent web search.
- n=6, single run — same caveats as Round 1: o200k_base proxy tokenizer, inference overhead in timings, no variance data.
Raw materials: ~/Developer/sandbox/deej/context7-token-test/pinned-vs-web/ — transcripts (context7/, websearch/), timing.log, analyze.py, results.json. Round 1 report: Context7 vs Web Search — Token & Speed Test.