When 84% Token Reduction Means Zero Accuracy
Semantic code search cut my token spend by 84% on a real Laravel debugging task and found zero of the four files that mattered. I had been evangelizing the tool for weeks before I finally benchmarked it.
The Setup
I had been evangelizing semantic code search for weeks. VectorCode promised to cut through large codebases by understanding meaning, not just matching strings. So I did what I should have done from the start: I ran a real benchmark.
The task was concrete: debug a payment processing bottleneck in a Laravel app. The files that had to be found were PaymentHubController at the entry point, GatewayPaymentHubService where the bottleneck lived, GatewayPayment with the N+1 query, and PaymentRepository underneath it. A realistic debugging scenario, not some toy example.
The Numbers That Stung
Vanilla Claude Code: 91,457 tokens, 38 tool calls, 142 seconds. Found all 4 critical files. 100% accuracy.
VectorCode: roughly 15,000 tokens, 6 queries, 8 seconds. Found zero critical files. 0% accuracy.
I stared at that for a while. An 84% token reduction sounds incredible until you realize it produced 100% false positives. You save nothing when the answer is wrong. It is like bragging about fuel efficiency on a car that drives to the wrong city.
Why It Failed
The failure modes were instructive. Semantic embeddings are great at matching documentation and test files, things written in natural language that describe what code does. But terse code? Variable names like $gatewayPayment and method signatures like processHubPayment()? The embeddings had no idea what to do with those. VectorCode matched conceptually adjacent files (docs about payments, test helpers) instead of the code that mattered.
Meanwhile, vanilla CC did what good debugging actually looks like. It followed the framework's structure: routes to controllers to services to repositories. Pattern-based discovery. Progressive refinement. It knew Laravel, so it knew where to look. No semantic magic needed, just methodical grep-and-read cycles.
What I Actually Learned
My first instinct was to throw VectorCode out entirely. But that felt reactionary. The tool is fine. I was aiming it at the wrong job. When I asked it conceptual questions like "how does payment processing work in this codebase?" it returned useful orientation results. It is a discovery tool, and I had been using it as a precision one.
The hybrid approach is the real answer. VectorCode for the "I just joined this project, where do I even start?" phase. Vanilla tools for "there is an N+1 query somewhere in the payment flow and I need to find it now."
Token efficiency is worth counting only when the answer stays right. An 84% reduction that produces garbage is waste with better optics.
I am glad I ran the benchmark before building my entire workflow around it. The numbers do not care about my enthusiasm.