# From a PagerDuty alert to the real checkout blast radius

A checkout service failed its readiness checks. PagerDuty reached the on-call engineer, and Anyshift showed what else could fail before a local fix became a wider checkout incident.

Author: Stephane Jourdan

Published: 2026-08-04T12:00:00-07:00

Source: https://anyshift.io/blog/anyshift-meets-pagerduty

## TL;DR

A checkout service failed its readiness checks. PagerDuty routed the alert to the on-call engineer, while Anyshift traced the production dependencies behind it. That context let the engineer assess the blast radius before remediation, reducing the risk of fixing one service while breaking another.

![Three-step workflow: PagerDuty identifies the checkout alert, Anyshift maps its production blast radius, and PagerDuty receives the failure and dependency context.](/images/blog/anyshift-meets-pagerduty-context-workflow.svg?v=3)

## What does the page tell the engineer, and what is missing?

Every incident raises two questions: **what failed, and what could fail with it?** [PagerDuty turns a monitoring signal into an incident](https://support.pagerduty.com/main/docs/alerts) and routes it through an [escalation policy and on-call schedule](https://support.pagerduty.com/main/docs/escalation-policies-and-schedules), answering the first question for the right engineer.

The second question defines the blast radius. During a [P1 incident](https://support.pagerduty.com/main/docs/incident-priority), restarting one pod could remove the last healthy endpoint from a shared path.

At Northstar Market, a fictional online retailer, `checkout-api` pods were running but failed their [readiness probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/) on port `8081`. Kubernetes stopped routing traffic, and PagerDuty paged the checkout team. The alert showed the failure, but not how much of checkout depended on it.

## What could fail with `checkout-api`?

Before changing anything, the on-call engineer followed four questions:

1. **Which production deployment is actually failing?** Confirm the exact deployment and environment named by the alert.
2. **Is this failure isolated or recurring?** Check whether the probe failure is part of a broader pattern.
3. **If I change this deployment, what else could be affected?** Trace its downstream blast radius before restarting or remediating it.
4. **Do other services share a datastore discovered on that path?** Check whether shared state extends the blast radius beyond direct dependencies.

These questions shape the order of the [Anyshift Graph SDK](https://docs.anyshift.io/pages/product/integration/sdk) investigation. It first resolves the production deployment, then checks its failure history and service tree. If the service tree reveals shared state, the integration follows that datastore next.

**Initial PagerDuty alert**

```json compact
{
  "component": "checkout-api",
  "signal": "readiness probe failed"
}
```

The PagerDuty alert identifies `checkout-api`, but the same name could exist in several environments or resource types. Anyshift matches it to the Kubernetes deployment running in the production checkout namespace. This gives every following query one exact starting point.

**Anyshift input: Which production deployment is actually failing?**

```bash compact question-accent sql-accent
# 1. Which production deployment is actually failing?
annie graph query \
  "SELECT * FROM resolve
   WHERE term = checkout-api
   LIMIT 10" \
  --output json
```

**Anyshift output: `checkout-api` resolves to one production deployment.**

```json compact
{
  "name": "checkout-api",
  "type": "K8S_DEPLOYMENT",
  "namespace": "checkout",
  "cluster": "anyshift-demo-eks"
}
```

**Explanation:** The result pins `checkout-api` to one deployment, namespace, and cluster, so later queries do not mix resources from different environments.

**Next step:** Northstar's on-call engineer now wants to know whether the failure is recurring and what a change to `checkout-api` could affect.

The CLI below shows those queries through the [Anyshift Graph Query Language](https://docs.anyshift.io/pages/product/integration/graph_query_language):

**Anyshift inputs: Is the failure recurring, and what could be affected?**

```bash compact question-accent sql-accent
# 2. Is this failure isolated or recurring?
annie graph query \
  "SELECT * FROM failures
   WHERE target = checkout-api
     AND namespace = checkout
     AND since = 90d
   LIMIT 100" \
  --output json

# 3. If I change checkout-api, what could be affected?
annie graph query \
  "SELECT * FROM servicetree
   WHERE target = checkout-api
     AND source = auto
   LIMIT 30" \
  --output json

```

**Anyshift output: The failure has recurred across 90 days.**

```json compact
{
  "failure_events": 53,
  "pod_unhealthy_events": 44,
  "pod_failed_scheduling_events": 9,
  "readiness_probe_8081_events": 4
}
```

**Explanation:** Anyshift found 53 failures over 90 days: 44 unhealthy-pod events and 9 scheduling failures. Four were readiness-probe connection failures on port `8081`. **This is a recurring pattern, so a restart alone may not resolve it.**

**Anyshift output: `checkout-api` reaches 12 services and two datastores.**

```json compact
{
  "downstream_services": 12,
  "datastores": ["redis-cart", "checkout-postgres"]
}
```

**Explanation:** The service tree identifies two datastores behind `checkout-api`: `redis-cart` and `checkout-postgres`. Because `checkout-postgres` may also serve workloads outside the direct call chain, the engineer next asks which other services depend on it.

**Anyshift input: Which other services share `checkout-postgres`?**

```bash compact question-accent sql-accent
# 4. Which other services share checkout-postgres?
annie graph query \
  "SELECT * FROM datastore
   WHERE target = checkout-postgres
     AND source = auto
   LIMIT 30" \
  --output json
```

**Anyshift output: Three services share `checkout-postgres`.**

```json compact
{
  "datastore": "checkout-postgres",
  "dependent_services": ["checkout-api", "order-creation-worker", "checkout"]
}
```

**Explanation:** `checkout-postgres` is shared by `checkout-api` and two other services: `order-creation-worker` and `checkout`. This does not prove that PostgreSQL caused the incident. It shows that a change affecting the shared datastore could affect all three services, so the engineer should not treat `checkout-api` as isolated.

Now that the four Anyshift queries have returned the incident context, a JavaScript function maps those values into PagerDuty's `custom_details` fields.

**Integration output: fields sent to PagerDuty (cropped)**

```json
{
  "affected_component": "checkout-api",
  "failure_events": 53,
  "readiness_probe_8081_events": 4,
  "downstream_services": 12,
  "datastores": 2,
  "shared_datastore": "checkout-postgres",
  "shared_datastore_services": 3
}
```

**Explanation:** `checkout-api` is not isolated: its path reaches 12 services and two datastores, while `checkout-postgres` is shared by three services.

## How the context lands in PagerDuty

The integration sends the Anyshift context through the [PagerDuty Events API v2](https://developer.pagerduty.com/docs/events-api-v2/trigger-events/) as native `custom_details`. The failure history, downstream footprint, and shared datastore stay attached to the incident that PagerDuty routes and manages.

The screenshot below shows the accepted alert with those values in PagerDuty [Custom Details](https://support.pagerduty.com/main/docs/pd-cef).

![The real PagerDuty alert shown at a smaller scale with annotations for the Anyshift source, blast radius, and failure evidence.](/images/blog/anyshift-meets-pagerduty-native-custom-details-annotated-v6.png?v=6)

How does this work without an engineer manually running SDK calls during a P1? In the automated path, Anyshift listens for an `incident.triggered` webhook through its [PagerDuty integration](https://docs.anyshift.io/pages/integration/pagerduty), runs the same investigation, and posts the [root-cause analysis](https://docs.anyshift.io/pages/product/root_cause_analysis) back as incident notes and into the mapped Slack thread.

## Which question does PagerDuty need answered next?

More graph data is not automatically more useful during an incident. **Each response step needs the smallest slice of context that can change its next action.**

```yaml compact
pagerduty_alert: [component, signal]
anyshift_graph: [cause, blast_radius, safety, recovery]
pagerduty_response: [route, investigate, automate, close]
```

| Moment | Question | Anyshift supplies | PagerDuty acts |
| --- | --- | --- | --- |
| Before the page | Who is affected, and how urgent is this? | Blast radius, SLO risk, and alert-noise context | [Event Orchestration](https://support.pagerduty.com/main/docs/event-orchestration) sets priority, routing, or suppression |
| During triage | What changed, and how did the failure spread? | Likely cause and the relevant dependency path | [SRE Agent](https://support.pagerduty.com/main/docs/sre-agent) investigates that production path |
| Before remediation | Could this action make the incident worse? | Disruption-budget, scaling, storage, and access checks | [Automation Actions](https://support.pagerduty.com/main/docs/automation-actions) runs after the safety check |
| Before closing | Has the service and its dependency path recovered? | Fresh alert, SLO, and audit evidence | Incident Workflows verify recovery and retain the evidence |

**Why make the context structured instead of adding a prose note?** PagerDuty supports [SRE Agent connectors through MCP or APIs](https://support.pagerduty.com/main/changelog/new-configuration-experience-sre-agent-connectors-and-tools-generally-available), while [external data variables](https://support.pagerduty.com/main/changelog/external-data-variables-are-generally-available-for-aiops-customers) can put API-supplied facts into orchestration rules. **Blast radius or SLO risk can then become a routing condition**, not just a note someone must remember to read.

The same **question-first contract** works when an alert points to a failing node, a risky deploy, a vulnerable image, or a shared Kafka topic. The Anyshift [Graph SDK capability guide](https://docs.anyshift.io/pages/product/integration/sdk_capabilities) lists the corresponding change, security, stream, topology, and Kubernetes safety queries.

The reusable principle is simple: **before changing the failing component, ask what else moves with it.** Then put that answer where the on-call engineer will make the decision.

**Using PagerDuty and facing an incident like this?** [Send it to us](/contact).