# In Our API Benchmark, Normalization Mattered More Than Model Size

We tested six models on a bounded task: turn a plain-English infrastructure question into one supported query. Deterministic normalization and execution erased most output differences, leaving Haiku and Luna slightly ahead of Opus on final-answer accuracy. Luna then matched Haiku at 21% of the observed cost.

Author: Stephane Jourdan

Published: 2026-08-12

Source: https://anyshift.io/blog/llm-routed-graph-api-evals

## TL;DR

- `/ask` maps a plain-English infrastructure question to one supported lookup. **The model chooses; application code normalizes, validates, and runs the request.**
- Across **6,336 requests**, exact raw-route accuracy was 42.71% to 52.94%. Final-answer accuracy reached 99.15% to 100%. **Normalization mattered more than model tier.**
- Haiku and Luna answered every request correctly; Opus missed three. **Luna matched Haiku at 21% of the observed cost**, then explicit prompt reuse cut Luna's cost by another **14.2%**.
- A later holdout on two additional production graphs was harder: **Luna reached 90.83% final-answer accuracy against Haiku's 88.33%, at about 20% of the cost.**

## This Was Not a Chatbot

Engineers want to ask, **“Which services depend on this database?”** They do not want to write query syntax first.

Anyshift's [Graph API](/blog/introducing-graph-api) reads a live map of a company's production infrastructure. Its `/ask` endpoint turns that plain-English question into one supported lookup. The model chooses the lookup and its parameters. Application code normalizes and validates the choice, then retrieves current evidence.

We were considering a move from Claude Haiku 4.5 to GPT-5.6 Luna. The benchmark focused on one question: **Would changing the model change the answer returned to users?**

Not by much. Once the query contract and normalization were in place, **higher model tiers did not improve the final answers**.

## The Infrastructure Questions We Tested

The endpoint supports questions from three common infrastructure workflows:

- **Change planning:** “Which services depend on checkout?” and “What is the blast radius if this database becomes unavailable?”
- **Incident investigation:** “What happened to `anyshift-backend` during the last seven days?” and “Which resources had the most changes today?”
- **Inventory and security:** “How many ECS services are in the graph?” and “What IAM role changes happened in the last seven days?”

These are evidence-retrieval tasks. **The model interprets the question. The infrastructure facts come from Anyshift.**

## The AI Chose a Lookup. Application Code Did the Rest.

The AI model selects one lookup and its parameters from a fixed menu. We call that selection a **route**. **The model never searches the infrastructure data or writes the final answer.**

![The AI model chooses a lookup; Anyshift checks it, runs it, and returns current infrastructure evidence](/images/blog/llm-graph-routing-pipeline.svg?v=6)

The model interprets the language. Application code owns the allowed lookups and repair logic, while Anyshift's live infrastructure map supplies the facts.

The endpoint maps one question to one operation in a fixed catalog. It handles ambiguity in labels, aliases, scopes, time windows, dependency direction, and parameters. **It does not plan a multi-step investigation or synthesize an open-ended answer.** Adding those questions would benchmark Annie, not `/ask`.

## How We Built the Benchmark

We built the benchmark from two live infrastructure maps. Each approved case contained one graph operation, its parameters, and the evidence it should return. We wrote four natural-language versions of each case and ran every question three times per model. **That produced 6,336 requests, scored against deterministic graph results with no judge model.**

| Benchmark control | Configuration |
| --- | --- |
| Questions | 352 approved questions from two anonymized live graphs |
| Test cases | 88, with four question variants per case |
| Repetitions | Three per question and model |
| Models | Haiku 4.5, Sonnet 5, Opus 5, Luna, Terra, and Sol |
| Total requests | 6,336 |
| Reference | Deterministic graph query for every question |
| Reasoning settings | OpenAI models at `none`; Sonnet and Opus at `low` with thinking disabled |

The source graphs could change, so we generated the deterministic reference at run time and kept model runs close to the same snapshots. This benchmark measures natural-language routing for one API, not general reasoning, coding, or agent performance.

## What We Measured at Each Step

To see where differences were corrected, we retained five versions of each request:

- **Raw model route:** the operation and parameters returned directly by the model.
- **Canonical route:** the human-approved operation and parameters used as the expected route.
- **Normalized route:** the model route after deterministic aliases and defaults were applied.
- **Effective graph route:** the operation actually executed after resource resolution.
- **Final answer:** the graph evidence returned to the user.

These layers answer different questions: **Did the model return the exact route? Did normalization repair it? Did the user receive the expected evidence?**

One retained Luna request against our own production graph shows why those distinctions matter:

| Layer | Observed result |
| --- | --- |
| Question | “How many ECS services are in the graph?” |
| Raw model route | `inventory(term: "ECS service")` |
| Canonical route | `inventory(term: "ECS_SERVICE")` |
| Normalized route | `inventory(term: "ECS_SERVICE")` |
| Effective graph route | `inventory(term: "AWS_ECS_SERVICE")` |
| Graph answer | `48 AWS_ECS_SERVICE resources across 1 namespace.` |

The model chose the correct inventory operation but used a natural-language label instead of the internal enum. The raw comparison failed; the normalizer repaired the label, resource resolution selected the AWS-specific type, and **the answer matched the reference**.

The same pattern appeared elsewhere. Haiku, Luna, and Terra each had 48 normalized routes that differed from the canonical route while returning the expected evidence. All 1,056 final answers from each model were correct.

**Raw-route accuracy measures the model in isolation. Final-answer accuracy measures the product the user called.** We retain every layer because a correct answer can still hide growing dependence on repair logic.

## Different Model Outputs, Nearly Identical Answers

Every model exceeded 99% final-answer accuracy. **Haiku 4.5, Luna, and Terra answered all 1,056 requests correctly.** Sonnet and Opus missed three answers each; Sol missed nine. The full spread was 0.85 percentage points.

![Accuracy, latency, and cost results for six models](/images/blog/llm-graph-routing-benchmark-results.svg?v=6)

The gap between model output and product output was much larger. Raw-route accuracy was **42.71% to 52.94%**. After aliases and defaults, it reached **94.60% to 95.45%**. Resource resolution and execution brought final-answer accuracy to **99.15% to 100%**.

Application code did not make every route identical. It made harmless differences such as `ECS service` versus `ECS_SERVICE` irrelevant to the returned evidence.

Haiku and Luna each returned 1,056 correct answers; Opus returned 1,053. **The higher-tier model provided no accuracy advantage on this task.**

## Luna Won the Operating Tradeoff, Not the Accuracy Test

Haiku and Luna both returned every answer correctly. Haiku was faster, with a 1.30-second median and 2.52-second p95. Luna's median was 1.47 seconds and its p95 was 3.69 seconds, but **Luna cost 79% less:** $0.255 per 1,000 requests against Haiku's $1.210.

A later paired study gave both models explicit caching. Their normalized routes had no disagreements across 1,056 matched requests. Luna's median was 949 ms against Haiku's 981 ms; Haiku kept the lower p95 at 2,041 ms against 2,380 ms. **Luna cost about 82% less.**

Across the same paired run, Haiku used 11.19 million total tokens and Luna used 10.13 million. Luna used **9.5% fewer tokens overall and 31.9% fewer output tokens**. Both models read 99.8% of their input tokens from cache, with nearly identical uncached input totals. Lower list prices drove most of the 82% cost gap. Because the providers tokenize the same prompt differently, these totals are billing evidence rather than a direct measure of how much work each model performed.

Without Luna's pricing and caching advantage, Haiku would have been a close choice. Luna matched its answer quality at a substantially lower observed cost, making **Luna the better fit for this high-volume endpoint**.

Opus and Sonnet did not improve the user-facing answers on this task. That is not a claim against either model or Anthropic. A higher-tier model was a poor fit for an endpoint where the model only selected one lookup and application code handled the rest.

## Reusing the Shared Prompt Cut Luna's Cost by 14.2%

`/ask` sends the same instructions and fixed menu with every question. Prompt caching lets the provider reuse that unchanged prefix instead of processing it again. An explicit cache boundary marks exactly where the shared part ends.

OpenAI explains this behavior in its [current model guidance](https://developers.openai.com/api/docs/guides/latest-model); Anthropic documents [automatic and explicit prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching).

We compared automatic caching with an explicit cache boundary on the same 1,056 Luna requests. Cache-read share is the portion of input tokens reused; cache-write share is the portion stored for later requests.

| Luna configuration | Normalized-route accuracy | Cache-read share | Cache-write share | Median latency | p95 latency | Cost per 1,000 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Automatic caching | 95.45% | 97.89% | 2.08% | 924 ms | 2,195 ms | $0.260 |
| Explicit cache boundary | 95.45% | 99.52% | 0.28% | 927 ms | 2,208 ms | $0.223 |

**Explicit caching reduced observed cost by 14.2% and cache writes by 86.4%.** Accuracy stayed fixed, and latency barely moved.

The cost calculations use the published standard rates available on August 10, 2026. See the current [OpenAI model pricing](https://developers.openai.com/api/docs/models/gpt-5.6-luna) and [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing) for updates.

## Two More Graphs Exposed Gaps in the Contract

We later ran Haiku and Luna against two additional anonymized production graphs in a separate holdout. Forty new cases, four question variants per case, and three repetitions produced **960 requests across 160 new questions**.

The first pass exposed two gaps in our application code. Provider-prefixed resource aliases were not normalized consistently, and a query for `DYNAMODB_TABLE` could also match `DYNAMODB_TABLESTREAM`. We fixed both before rerunning the full holdout.

| Model | Initial final-answer accuracy | Post-fix final-answer accuracy | Router median | Router p95 | Post-fix cost per 1,000 |
| --- | ---: | ---: | ---: | ---: | ---: |
| Haiku 4.5 | 83.75% | 88.33% | 1,019 ms | 1,663 ms | $1.238 |
| Luna | 87.92% | 90.83% | 1,033 ms | 3,263 ms | $0.245 |

The post-fix run returned HTTP 200 for all 960 requests. Luna answered 436 of its 480 requests correctly; Haiku answered 424. Luna cost **80.25% less**, while Haiku kept the better tail latency.

The cloud-resource cases give us the clearest regression signal. Each model returned the expected normalized route, effective route, and final answer on all 36 executions. A question about AWS DynamoDB tables now resolves to `DYNAMODB_TABLE` without including the two `DYNAMODB_TABLESTREAM` resources present in the same graph.

The remaining misses involved blast radius, impact, inventory, event windows, and resource resolution. Because these are live graphs and model routes can vary between runs, we cannot attribute every accuracy-point change to the two code fixes. On the unfamiliar graphs, **Luna kept a 2.5-point final-answer advantage at roughly one-fifth of Haiku's observed model cost.**

This later result sits alongside the six-model benchmark above. It shows what changed when we tested the same contract on two graphs that had not shaped the original dataset. The customer questions, graph identifiers, and resource values remain private.

## The Boundary of This Result

Cost uses measured token classes and the published rates captured on August 10, 2026. It excludes discounts, taxes, infrastructure overhead, and graph execution. Latency came from matched requests on one laptop, so **it is a system measurement, not a provider-wide comparison**.

The public [sanitized aggregate dataset](/data/llm-graph-routing-benchmark-2026-08-10.json) includes the original benchmark's full model totals, caching study, production timing, methodology, and claim boundaries. Customer names, questions, graph identifiers, and resource values are excluded.

The benchmark does not compare general reasoning, coding, or open-ended infrastructure investigation. **It only establishes that all six tested models produced nearly the same final-answer quality for the current `/ask` contract.**

## What We Changed

We switched `/ask` to GPT-5.6 Luna in standard mode and marked the shared prompt for reuse. **The public API did not change.** A production verification run completed 120 of 120 requests with correct routes and answers, no retries, and a 99.8% cache-read share.

Luna delivered Haiku's answer quality at 21% of the observed cost, while explicit caching reduced Luna's cost by another 14.2%. Haiku remains a sensible choice when tighter tail latency matters more. We kept both providers behind the same contract, so the model remains an operational choice rather than an API redesign.

## What did the tested Anyshift feature do?

It turned a natural-language infrastructure question into one lookup from a fixed menu, ran that lookup against current production data, and returned the evidence. The AI model selected the lookup and its parameters; application code checked and executed it.

## What did the Anyshift model benchmark test?

It tested six models on 352 approved natural-language questions derived from two live infrastructure graphs. Each question ran three times per model, producing 6,336 API executions.

## Which models did Anyshift compare?

The benchmark compared Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5, GPT-5.6 Luna, GPT-5.6 Terra, and GPT-5.6 Sol.

## Which model won the benchmark?

No model won every metric. Haiku 4.5, Luna, and Terra all produced 100% correct final answers. Haiku had the best broad-run latency, while Luna cost 79% less than Haiku. Reusing the shared prompt cut Luna's cost by another 14.2%. We chose Luna because that cost advantage mattered more than its modest broad-run latency tradeoff for this endpoint.

## Why were final-answer scores so close across models?

The AI model only selected a lookup and its parameters from a fixed menu. Application code standardized labels, filled in defaults, validated the request, and retrieved the evidence. Raw routes matched the exact reference in 42.71% to 52.94% of requests, while final-answer accuracy reached 99.15% to 100%. This result applies to that bounded task, not to open-ended reasoning.

## How did reusing the shared prompt affect GPT-5.6 Luna?

We marked the unchanged part of the prompt so the provider could reuse it instead of processing it again. In a matched 1,056-request study, this reduced observed cost by 14.2% and cut cache writes by 86.4%, with no accuracy change. Typical response time and the slowest 5% of requests were effectively unchanged.

## Did the models generalize to other infrastructure graphs?

A later holdout used 160 new questions from two additional anonymized production graphs. Luna returned the expected final answer in 90.83% of 480 requests, compared with 88.33% for Haiku, at about 20% of Haiku's observed model cost. The holdout also exposed deterministic alias and exact-type matching gaps that the original benchmark had not covered.