Investigate PagerDuty alerts. Get findings in Slack.

Annie investigates PagerDuty alerts using Kubernetes state, Datadog signals and recent code or infrastructure changes. Get evidence and next steps in Slack.

Before starting

  • Provide the alert, service and time window. Reuse the project’s infrastructure and observability connections. PagerDuty is needed for its automatic trigger; Slack is needed only if you choose Slack delivery.

Workflow

  • PagerDuty reports checkout-api errors after a rollout. The pods are ready, but requests fail. Ask Annie to compare the change with the application’s health signals.
  1. A PagerDuty incident triggers Annie according to your project’s automation rules.
  2. Annie compares Kubernetes state, Datadog signals and relevant changes in the incident window.
  3. The configured Slack channel receives findings, supporting evidence, missing data and next steps.

Set up

  • Connect the evidence: Reuse the project’s Kubernetes, Datadog and relevant repository integrations. Limit access to the resources needed for the investigation.
  • Choose which alerts start an investigation: Configure the PagerDuty webhook and its service scope, then set the project’s automation rules.
  • Choose the Slack destination: In Webhook investigation notifications, enable Default notify channel and select a channel mapped to the project.

Evidence

  • PagerDuty: An incident.triggered webhook can start an investigation. Project automation rules decide which events Annie investigates.
  • Kubernetes: The in-cluster integration supplies workloads and resource state. Helm-created resources are visible; chart versions and Helm revision history are not modeled as release entities.
  • Datadog: Logs, metrics, monitors and traces require their corresponding application-key scopes. A successful connection check alone does not grant access to all signals.
  • GitHub: Connect the relevant repositories when source code or commit context is needed. Code access is separate from the Kubernetes and monitoring connections.
  • Slack: Map a channel to the project and enable Default notify channel for PagerDuty webhook investigations.
  • AWS changes: If the investigation needs AWS API activity, CloudTrail forwarding supplies that evidence. Reuse it when already configured.

Expected result

  • A connection-pool change may explain database timeouts if the evidence supports it. Ready pods alone do not prove the application is healthy.
  • Annie can only assess data available in the selected project and incident window. Missing permissions, unavailable telemetry or gaps in history must remain explicit. Nearby deployment and alert timestamps alone do not prove a root cause.

Can Annie investigate without an automatic trigger?

  • Yes. Ask Annie in a mapped Slack channel, use the web app, or connect Annie Remote MCP from your coding assistant. Include the alert, affected service and time window.

What happens when evidence is missing?

  • The result should identify unavailable signals and unresolved questions. Ready pods or a nearby deployment alone do not prove recovery or a root cause.

Does an investigation change production?

  • Cloud and Kubernetes evidence uses read-only access. Integration actions are separate: the PagerDuty integration documents acknowledgements and comments, and Slack delivery posts messages. Check each integration’s scopes; a prompt is not a substitute for permission controls.

Investigate on demand with Annie Remote MCP

In the intended Anyshift project, investigate [alert] for [service / environment] during [time window]. Confirm the project with list_projects. Use ask_annie with mode=rca, then poll the same investigation with get_annie_response until it completes or fails. Compare deployment changes with health signals. Return evidence, likely causes, missing data and the next check. Do not change infrastructure or send messages.

Usage

  • An investigation run by Anyshift uses the agent run allowance. Use Graph MCP if your own agent only needs to retrieve evidence.