# Investigate PagerDuty alerts. Get findings in Slack.

Annie investigates PagerDuty alerts using Kubernetes state, Datadog signals and recent code or infrastructure changes. Get evidence and next steps in Slack.

Source: https://anyshift.io/for-agents/use-cases/investigate-deployment-alert

## Before starting

- Provide the alert, service and time window. Reuse the project’s infrastructure and observability connections. PagerDuty is needed for its automatic trigger; Slack is needed only if you choose Slack delivery.

[Set up PagerDuty investigations](https://docs.anyshift.io/pages/product/root_cause_analysis)

## Workflow

- PagerDuty reports checkout-api errors after a rollout. The pods are ready, but requests fail. Ask Annie to compare the change with the application’s health signals.

1. A PagerDuty incident triggers Annie according to your project’s automation rules.

2. Annie compares Kubernetes state, Datadog signals and relevant changes in the incident window.

3. The configured Slack channel receives findings, supporting evidence, missing data and next steps.

## Set up

- Connect the evidence: Reuse the project’s Kubernetes, Datadog and relevant repository integrations. Limit access to the resources needed for the investigation.

- Connect PagerDuty: Create an API key in PagerDuty → Integrations → API Access Keys and save it in Anyshift → Integrations → PagerDuty → New Token. Reuse an existing token connection.

- Register the trigger: In Anyshift → Integrations → PagerDuty → New Webhook, choose a name and secret. Copy the URL and secret to PagerDuty → Integrations → Generic Webhooks (V3) → New Webhook. Select the service scope, include incident.triggered, and set Signature Secret to the same secret.

- Check investigation routing: The router can start an RCA, open a chat or skip an event. Review the project’s Automation rules and verify the selected service’s behavior with a test event.

- Choose the Slack destination: Install Annie in Slack, invite it to a channel and map that channel to the same Anyshift project. In Integrations → Slack → Webhook investigation notifications, enable notifications, choose start/complete/fail events, then enable Default notify channel and select the mapped channel. This destination is off by default.

- Verify delivery: With approval to create an incident and post to the chosen channel, trigger one test incident in the selected PagerDuty service. Confirm it appears in Anyshift, the expected automation runs, and the completed findings reach Slack. A started run alone is not a successful delivery.

[Connect the evidence](https://docs.anyshift.io/pages/onboarding/overview_integrations)

[Connect PagerDuty](https://docs.anyshift.io/pages/integration/pagerduty#step-1-add-an-api-token)

[Register the trigger](https://docs.anyshift.io/pages/integration/pagerduty#step-2-configure-the-webhook)

[Check investigation routing](https://docs.anyshift.io/pages/product/customization/instructions)

[Choose the Slack destination](https://docs.anyshift.io/pages/product/integration/slack#other-settings)

[Verify delivery](https://docs.anyshift.io/pages/integration/pagerduty#verify-the-connection)

## Evidence

- PagerDuty: An incident.triggered webhook can start an investigation. Project automation rules decide which events Annie investigates.

- Kubernetes: The in-cluster integration supplies workloads and resource state. Helm-created resources are visible; chart versions and Helm revision history are not modeled as release entities.

- Datadog: Logs, metrics, monitors and traces require their corresponding application-key scopes. A successful connection check alone does not grant access to all signals.

- GitHub: Connect the relevant repositories when source code or commit context is needed. Code access is separate from the Kubernetes and monitoring connections.

- Slack: Map a channel to the project and enable Default notify channel for PagerDuty webhook investigations.

- AWS changes: If the investigation needs AWS API activity, CloudTrail forwarding supplies that evidence. Reuse it when already configured.

[PagerDuty](https://docs.anyshift.io/pages/integration/pagerduty#what-happens-when-an-incident-fires)

[Kubernetes](https://docs.anyshift.io/pages/iac/kubernetes_live)

[Datadog](https://docs.anyshift.io/pages/integration/datadog#4-required-permissions)

[GitHub](https://docs.anyshift.io/pages/integration/github)

[Slack](https://docs.anyshift.io/pages/product/integration/slack#other-settings)

[AWS changes](https://docs.anyshift.io/pages/integration/aws/cloudtrail)

## Expected result

- A connection-pool change may explain database timeouts if the evidence supports it. Ready pods alone do not prove the application is healthy.

- Annie can only assess data available in the selected project and incident window. Missing permissions, unavailable telemetry or gaps in history must remain explicit. Nearby deployment and alert timestamps alone do not prove a root cause.

## Compass Digital ruled out an unrelated change

- In Anyshift’s published Compass Digital case, a sign-in issue followed a CI/CD identity change. Annie traced the sign-in dependency path to shared SSO and ruled out the unrelated change. Incident-time telemetry located the interruption; the SSO owner confirmed an outage during an upgrade.

- The decision was to avoid an unrelated rollback. This is a customer case published by Anyshift, not a controlled performance comparison. A nearby change alone did not establish the cause.

[Read the Compass Digital case (Markdown)](https://anyshift.io/markdown/blog/compass-digital-login-outage.md)

## Can Annie investigate without an automatic trigger?

- Yes. Ask Annie in a mapped Slack channel, use the web app, or connect Annie Remote MCP from your coding assistant. Include the alert, affected service and time window.

## What happens when evidence is missing?

- The result should identify unavailable signals and unresolved questions. Ready pods or a nearby deployment alone do not prove recovery or a root cause.

## Does an investigation change production?

- Cloud and Kubernetes evidence uses read-only access. Integration actions are separate: the PagerDuty integration documents acknowledgements and comments, and Slack delivery posts messages. Check each integration’s scopes; a prompt is not a substitute for permission controls.

## Investigate on demand with Annie Remote MCP

```text
In the intended Anyshift project, investigate [alert] for [service / environment] during [time window]. Confirm the project with list_projects. Use ask_annie with mode=rca, then poll the same investigation with get_annie_response until it completes or fails. Compare deployment changes with health signals. Return evidence, likely causes, missing data and the next check. Do not change infrastructure or send messages.
```

[Connection instructions](https://docs.anyshift.io/pages/product/integration/remote_mcp#setup)

[Read the investigation guide](https://docs.anyshift.io/pages/product/root_cause_analysis#investigate-a-deployment-alert)

## Usage

- An investigation run by Anyshift uses the agent run allowance. Use Graph MCP if your own agent only needs to retrieve evidence.

[Pricing](https://anyshift.io/for-agents/pricing)