AIOps Incident-Response Agent

Q: How do you handle 200 alerts in 60 seconds (alert storm)?

Dependency-graph correlation collapses them into one incident. Investigate the root node (first service to degrade), not all 200 downstream effects.

Q: What if RCA is wrong and the fix makes things worse?

Circuit breaker in verification step detects worsening golden signals → halts + escalates. Never execute a second remediation without human confirmation.

Q: How do you prevent LLM hallucinating a root cause?

Require evidence citation - LLM must quote a specific log line or metric spike. If it can't cite evidence, hypothesis is rejected. Cross-check against recent deploys.

Q: What about prompt injection via a crafted log line?

Log content goes in a clearly delimited data section, never in the instruction section. Action execution only accepts structured action specs - never free-form LLM text.

Q: How do you build trust with skeptical on-call engineers?

Start in observe-only mode. Agent proposes, humans act. Graduate to auto-execute for one narrow class (pod restarts) with a kill switch. Expand based on track record.

Q: What if the on-call is unresponsive and the fix is medium-risk?

Escalate up the PagerDuty chain. Never expand autonomous permissions because a human is slow to respond.

Design an agent that receives production alerts, investigates root cause, and executes or proposes a fix - without making things worse.

Last updated: September 2026

An agent that receives production alerts, investigates root cause, and proposes or executes a fix - without making things worse. The agent ingests alerts from PagerDuty or Prometheus, gathers context from metrics, logs, and traces in parallel, runs root-cause analysis using an LLM with runbook retrieval, then either auto-executes low-risk remediations or routes higher-risk actions for human approval.

“Cost of inaction is far less than cost of wrong action. When uncertain - investigate and propose, never guess and execute. Confidence threshold for auto-execute is deliberately high at 0.85.”

Clarifying Questions (Ask These First ~5 min)

Question	Why it matters
What generates alerts? (PagerDuty, Prometheus, Datadog)	Determines ingestion layer
Investigate-only or autonomous execution?	Most critical - sets blast radius
Do runbooks exist?	Changes RCA approach entirely
What observability is available? (metrics/logs/traces)	Limits what signals agent can use
Single-process or distributed fleet?	Affects alert correlation complexity
When to escalate to human?	Defines confidence thresholds

Architecture (Draw This)

Alert (PagerDuty/Prometheus)
  → [1] Ingest & Deduplicate
  → [2] Gather Context (parallel: metrics, logs, traces, recent deploys)
  → [3] RCA  (LLM + runbook retrieval + ReAct loop)
  → [4] Remediation Plan  (action spec + safety classification)
       ├── confidence > 0.85 + LOW-RISK  →  [5] Auto-Execute
       └── else                          →  [6] Human Approval (Slack)
  → [7] Verify (poll metrics 30s × 10)
  → [8] Postmortem draft + runbook update

Key Components (30-second pitch each)

1Ingest & Deduplicate - Group alerts firing within 60s on the same service into one incident. Don't investigate each of 200 cascade alerts individually - find the root service using the dependency graph.
2Context Gathering (parallel fan-out) - Simultaneously fetch: alerting metric ±2hr, ERROR logs ±10min, recent deploys in past 2hr, K8s events, similar past incidents (vector search). Time budget: 30-90s.
3Root Cause Analysis (RCA) - Prompt LLM with all signals + matching runbook. Require chain-of-thought + confidence score. If confidence < 0.7 → escalate to human, don't act. Allow up to 3 ReAct follow-up queries if more signals needed.
4Execution Controls - Dry-run before any non-read action. One action at a time - verify before next. Circuit breaker: if golden signals worsen post-action → halt + escalate immediately.
5Verification - Poll alerting metric every 30s for 5min. Check 4 golden signals (latency, error rate, traffic, saturation). Outcomes: RESOLVED / PARTIAL / NO_CHANGE / WORSENED.

Safety Policy - The Most Important Table

Action Class	Example	Policy
Read-only	Query metrics	Always auto
Low-risk reversible	Restart pod, scale replicas	Auto if confidence > 0.85
Medium-risk	Rollback deployment	Human approval
High-risk	Delete resource, change ACL	Incident commander only
NEVER	Drop table, modify IAM	Hard block

MTTR Target

Path	Time
Alert → agent triggered	< 30s
Context gathering	30-90s
RCA	30-60s
Auto-execute path (total)	~5-10 min
Human-approval path (total)	~10-20 min
Human on-call baseline	30-90 min

3 Biggest Risks

1Wrong RCA → wrong fix - Mitigated by confidence gating + evidence citation - LLM must quote a specific log line or metric spike, otherwise hypothesis is rejected.
2Alert storm mishandling - Mitigated by dependency-graph correlation that collapses 200 cascade alerts into one incident at the root service.
3Agent making incident unrecoverable - Mitigated by safety policy table + circuit breaker that halts and escalates if golden signals worsen after any action.

Google Stack: Cloud Monitoring + Cloud Logging + Cloud Trace → Pub/Sub (ingestion) → Vertex AI / Gemini (RCA) → GKE (execution) → Vertex AI Vector Search (runbook retrieval) → BigQuery (audit log + postmortem corpus)

Azure Stack: Azure Monitor + Log Analytics + Application Insights → Event Hubs (ingestion) → Azure SRE Agent (GA March 2026, autonomous incident investigation) → Azure OpenAI GPT-4o (RCA) → AKS (execution) → Azure AI Search (runbook retrieval) → Microsoft Sentinel (anomaly detection) → Log Analytics Workspace (audit log)

AWS Stack: CloudWatch + CloudWatch Logs + X-Ray → EventBridge (ingestion) → AWS DevOps Agent (GA March 2026, 94% root cause accuracy, 75% lower MTTR) → Amazon Bedrock / Claude (RCA) → EKS (execution) → Amazon OpenSearch (runbook retrieval) → AWS Systems Manager Automation (execution) → S3 + CloudTrail (audit log)

Other Options: Datadog AIOps + OpenAI (vendor-neutral, strong third-party integrations) | PagerDuty Operations Cloud + LLM layer | Dynatrace Davis AI (embedded AIOps, no-code) | Grafana Incident + LangChain (open-source path)

Frequently asked questions

How do you handle 200 alerts in 60 seconds (alert storm)?

Dependency-graph correlation collapses them into one incident. Investigate the root node (first service to degrade), not all 200 downstream effects.

What if RCA is wrong and the fix makes things worse?

Circuit breaker in verification step detects worsening golden signals → halts + escalates. Never execute a second remediation without human confirmation.

How do you prevent LLM hallucinating a root cause?

Require evidence citation - LLM must quote a specific log line or metric spike. If it can't cite evidence, hypothesis is rejected. Cross-check against recent deploys.

What about prompt injection via a crafted log line?

Log content goes in a clearly delimited data section, never in the instruction section. Action execution only accepts structured action specs - never free-form LLM text.

How do you build trust with skeptical on-call engineers?

Start in observe-only mode. Agent proposes, humans act. Graduate to auto-execute for one narrow class (pod restarts) with a kill switch. Expand based on track record.

What if the on-call is unresponsive and the fix is medium-risk?

Escalate up the PagerDuty chain. Never expand autonomous permissions because a human is slow to respond.