In Our API Benchmark, Normalization Mattered More Than Model Size

TL;DR

• /ask maps a plain-English infrastructure question to one supported lookup. The model chooses; application code normalizes, validates, and runs the request.
• Across 6,336 requests, exact raw-route accuracy was 42.71% to 52.94%. Final-answer accuracy reached 99.15% to 100%. Normalization mattered more than model tier.
• Haiku and Luna answered every request correctly; Opus missed three. Luna matched Haiku at 21% of the observed cost, then explicit prompt reuse cut Luna's cost by another 14.2%.
• A later holdout on two additional production graphs was harder: Luna reached 90.83% final-answer accuracy against Haiku's 88.33%, at about 20% of the cost.

This Was Not a Chatbot

Engineers want to ask, “Which services depend on this database?” They do not want to write query syntax first.

Anyshift's Graph API reads a live map of a company's production infrastructure. Its /ask endpoint turns that plain-English question into one supported lookup. The model chooses the lookup and its parameters. Application code normalizes and validates the choice, then retrieves current evidence.

We were considering a move from Claude Haiku 4.5 to GPT-5.6 Luna. The benchmark focused on one question: Would changing the model change the answer returned to users?

Not by much. Once the query contract and normalization were in place, higher model tiers did not improve the final answers.

The Infrastructure Questions We Tested

The endpoint supports questions from three common infrastructure workflows:

• Change planning: “Which services depend on checkout?” and “What is the blast radius if this database becomes unavailable?”
• Incident investigation: “What happened to anyshift-backend during the last seven days?” and “Which resources had the most changes today?”
• Inventory and security: “How many ECS services are in the graph?” and “What IAM role changes happened in the last seven days?”

These are evidence-retrieval tasks. The model interprets the question. The infrastructure facts come from Anyshift.

The AI Chose a Lookup. Application Code Did the Rest.

The AI model selects one lookup and its parameters from a fixed menu. We call that selection a route. The model never searches the infrastructure data or writes the final answer.

The model interprets the language. Application code owns the allowed lookups and repair logic, while Anyshift's live infrastructure map supplies the facts.

The endpoint maps one question to one operation in a fixed catalog. It handles ambiguity in labels, aliases, scopes, time windows, dependency direction, and parameters. It does not plan a multi-step investigation or synthesize an open-ended answer. Adding those questions would benchmark Annie, not /ask.

How We Built the Benchmark

We built the benchmark from two live infrastructure maps. Each approved case contained one graph operation, its parameters, and the evidence it should return. We wrote four natural-language versions of each case and ran every question three times per model. That produced 6,336 requests, scored against deterministic graph results with no judge model.

The source graphs could change, so we generated the deterministic reference at run time and kept model runs close to the same snapshots. This benchmark measures natural-language routing for one API, not general reasoning, coding, or agent performance.

What We Measured at Each Step

To see where differences were corrected, we retained five versions of each request:

• Raw model route: the operation and parameters returned directly by the model.
• Canonical route: the human-approved operation and parameters used as the expected route.
• Normalized route: the model route after deterministic aliases and defaults were applied.
• Effective graph route: the operation actually executed after resource resolution.
• Final answer: the graph evidence returned to the user.

These layers answer different questions: Did the model return the exact route? Did normalization repair it? Did the user receive the expected evidence?

One retained Luna request against our own production graph shows why those distinctions matter:

The model chose the correct inventory operation but used a natural-language label instead of the internal enum. The raw comparison failed; the normalizer repaired the label, resource resolution selected the AWS-specific type, and the answer matched the reference.

The same pattern appeared elsewhere. Haiku, Luna, and Terra each had 48 normalized routes that differed from the canonical route while returning the expected evidence. All 1,056 final answers from each model were correct.

Raw-route accuracy measures the model in isolation. Final-answer accuracy measures the product the user called. We retain every layer because a correct answer can still hide growing dependence on repair logic.

Different Model Outputs, Nearly Identical Answers

Every model exceeded 99% final-answer accuracy. Haiku 4.5, Luna, and Terra answered all 1,056 requests correctly. Sonnet and Opus missed three answers each; Sol missed nine. The full spread was 0.85 percentage points.

The gap between model output and product output was much larger. Raw-route accuracy was 42.71% to 52.94%. After aliases and defaults, it reached 94.60% to 95.45%. Resource resolution and execution brought final-answer accuracy to 99.15% to 100%.

Application code did not make every route identical. It made harmless differences such as ECS service versus ECS_SERVICE irrelevant to the returned evidence.

Haiku and Luna each returned 1,056 correct answers; Opus returned 1,053. The higher-tier model provided no accuracy advantage on this task.

Luna Won the Operating Tradeoff, Not the Accuracy Test

Haiku and Luna both returned every answer correctly. Haiku was faster, with a 1.30-second median and 2.52-second p95. Luna's median was 1.47 seconds and its p95 was 3.69 seconds, but Luna cost 79% less: $0.255 per 1,000 requests against Haiku's $1.210.

A later paired study gave both models explicit caching. Their normalized routes had no disagreements across 1,056 matched requests. Luna's median was 949 ms against Haiku's 981 ms; Haiku kept the lower p95 at 2,041 ms against 2,380 ms. Luna cost about 82% less.

Across the same paired run, Haiku used 11.19 million total tokens and Luna used 10.13 million. Luna used 9.5% fewer tokens overall and 31.9% fewer output tokens. Both models read 99.8% of their input tokens from cache, with nearly identical uncached input totals. Lower list prices drove most of the 82% cost gap. Because the providers tokenize the same prompt differently, these totals are billing evidence rather than a direct measure of how much work each model performed.

Without Luna's pricing and caching advantage, Haiku would have been a close choice. Luna matched its answer quality at a substantially lower observed cost, making Luna the better fit for this high-volume endpoint.

Opus and Sonnet did not improve the user-facing answers on this task. That is not a claim against either model or Anthropic. A higher-tier model was a poor fit for an endpoint where the model only selected one lookup and application code handled the rest.

Reusing the Shared Prompt Cut Luna's Cost by 14.2%

/ask sends the same instructions and fixed menu with every question. Prompt caching lets the provider reuse that unchanged prefix instead of processing it again. An explicit cache boundary marks exactly where the shared part ends.

OpenAI explains this behavior in its current model guidance; Anthropic documents automatic and explicit prompt caching.

We compared automatic caching with an explicit cache boundary on the same 1,056 Luna requests. Cache-read share is the portion of input tokens reused; cache-write share is the portion stored for later requests.

Explicit caching reduced observed cost by 14.2% and cache writes by 86.4%. Accuracy stayed fixed, and latency barely moved.

The cost calculations use the published standard rates available on August 10, 2026. See the current OpenAI model pricing and Anthropic pricing for updates.

Two More Graphs Exposed Gaps in the Contract

We later ran Haiku and Luna against two additional anonymized production graphs in a separate holdout. Forty new cases, four question variants per case, and three repetitions produced 960 requests across 160 new questions.

The first pass exposed two gaps in our application code. Provider-prefixed resource aliases were not normalized consistently, and a query for DYNAMODB_TABLE could also match DYNAMODB_TABLESTREAM. We fixed both before rerunning the full holdout.

The post-fix run returned HTTP 200 for all 960 requests. Luna answered 436 of its 480 requests correctly; Haiku answered 424. Luna cost 80.25% less, while Haiku kept the better tail latency.

The cloud-resource cases give us the clearest regression signal. Each model returned the expected normalized route, effective route, and final answer on all 36 executions. A question about AWS DynamoDB tables now resolves to DYNAMODB_TABLE without including the two DYNAMODB_TABLESTREAM resources present in the same graph.

The remaining misses involved blast radius, impact, inventory, event windows, and resource resolution. Because these are live graphs and model routes can vary between runs, we cannot attribute every accuracy-point change to the two code fixes. On the unfamiliar graphs, Luna kept a 2.5-point final-answer advantage at roughly one-fifth of Haiku's observed model cost.

This later result sits alongside the six-model benchmark above. It shows what changed when we tested the same contract on two graphs that had not shaped the original dataset. The customer questions, graph identifiers, and resource values remain private.

The Boundary of This Result

Cost uses measured token classes and the published rates captured on August 10, 2026. It excludes discounts, taxes, infrastructure overhead, and graph execution. Latency came from matched requests on one laptop, so it is a system measurement, not a provider-wide comparison.

The public sanitized aggregate dataset includes the original benchmark's full model totals, caching study, production timing, methodology, and claim boundaries. Customer names, questions, graph identifiers, and resource values are excluded.

The benchmark does not compare general reasoning, coding, or open-ended infrastructure investigation. It only establishes that all six tested models produced nearly the same final-answer quality for the current /ask contract.

What We Changed

We switched /ask to GPT-5.6 Luna in standard mode and marked the shared prompt for reuse. The public API did not change. A production verification run completed 120 of 120 requests with correct routes and answers, no retries, and a 99.8% cache-read share.

Luna delivered Haiku's answer quality at 21% of the observed cost, while explicit caching reduced Luna's cost by another 14.2%. Haiku remains a sensible choice when tighter tail latency matters more. We kept both providers behind the same contract, so the model remains an operational choice rather than an API redesign.