Composing the Response: Building an Incident Pipeline from Scratch

Composing the Response: Building an Incident Pipeline from Scratch

Speakers: Geet Pradhan, Security Engineer

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=mFzsarGLOT4

Overview

In his compelling BSides SF talk, "Composing the Response: Building an Incident Pipeline from Scratch," security engineer Geet Pradhan addresses the critical challenges faced by lean security teams – often employee number one or two – operating with minimal budgets. Pradhan's presentation provides a pragmatic, experience-driven roadmap for developing robust incident response (IR) pipelines that enable organizations to transition from reactive panic to structured, efficient defense. The core premise revolves around the invaluable currency of time in cybersecurity incidents and how strategic automation can effectively "buy" precious minutes, fundamentally altering the outcome of an attack.

The talk builds upon Pradhan's previous BSides presentation on detection pipelines, extending the focus from identifying high-fidelity signals to creating an actionable "shield" that converts these signals into executable defensive measures. He candidly shares insights derived from his own experiences, highlighting both successes and pitfalls in building internal and external IR programs from the ground up. This article delves into his proposed five pillars of incident response, the crucial role of automation, the nuanced application of AI, and practical strategies for maturing an IR capability, even within resource-constrained environments.

For security professionals tasked with establishing or refining their incident response capabilities under tight constraints, Pradhan's framework offers a vital guide. His emphasis on foundational elements, smart automation, and continuous improvement provides a clear path to constructing an effective IR pipeline that safeguards both technical assets and the company's reputation and compliance standing. The article aims to unpack these strategies, offering a detailed technical exploration of how to compose a resilient and responsive security posture.

Background

▶ Watch: Introduction and target audience for the talk (0:00)

The landscape of cyber defense is often characterized by a stark imbalance: attackers leverage sophisticated automation, while defenders frequently grapple with manual, improvisation-heavy processes. Geet Pradhan vividly illustrates this disparity, describing the typical, chaotic response to an alert: initial panic, followed by a laborious, manual data collection process involving checking user accounts, device information from CMDBs or spreadsheets, IP intelligence, and then attempting to contact application owners who might be unavailable due to time zone differences or coffee breaks. This ad-hoc approach, which Pradhan terms the "improvisation tax," adds critical minutes and hours to incident resolution, directly benefiting the attacker.

Pradhan underscores that cyber incident response is fundamentally a race against time. The goal is to "buy time" to shift from a reactive state to a methodical, informed response. This problem is exacerbated in smaller organizations where security teams are lean, resources are scarce, and individuals often wear multiple hats. Without a structured pipeline, every lookup, every communication attempt, and every decision-making step contributes to this tax, making manual response inherently unscalable against an automated adversary. His previous talk focused on the "detection pipeline," aiming to distill high-fidelity signals from the vast noise of security logs. This current presentation picks up where that left off, focusing on how to convert those signals into decisive actions, forming the "shield" necessary for effective defense.

Key Findings

▶ Watch: Understanding the 'Improvisation Tax' in incident response (3:20)

Geet Pradhan's talk introduces a pragmatic framework centered around five pillars of incident response, designed to guide resource-constrained teams in building an effective pipeline. These pillars are:

1. Detection: Emphasizing the need to normalize chaotic alert sources into a single, manageable view.
2. Command Structure: Defining clear roles and responsibilities, with a strict rule of a single incident commander per event.
3. Coordination: Automating the establishment of a "war room" and centralizing all incident-related communications.
4. Execution: Prioritizing rapid containment of threats, moving towards automated containment with appropriate guardrails.
5. Governance & Preservation: Addressing the critical, often overlooked, aspects of compliance, legal obligations, and forensic evidence collection.

A central theme across these pillars is the paramount importance of time savings through automation. Pradhan argues that every minute saved in the initial stages of an incident allows defenders to move from a panic-driven reaction to a strategic, considered response. This includes automating initial triage, enrichment, and even containment for non-critical assets.

Another key finding is the distinction between auto-containment and auto-remediation. While auto-containment (stopping the bleeding) is achievable early in the maturity model, full auto-remediation (returning to normal operations) is significantly more complex and often requires a deeper understanding of the "normal" state, which can be elusive, especially for nascent security programs.

Finally, Pradhan provides a nuanced perspective on the role of Artificial Intelligence (AI) in incident response. He identifies specific "sweet spots" where AI can significantly enhance efficiency, such as alert normalization, context enrichment, and investigative assistance. However, he strongly cautions against blindly trusting AI for fully autonomous containment, legal/compliance decisions, or evidence handling, advocating for a human-in-the-loop approach in these critical areas to ensure safety and accountability.

Technical Deep Dive

▶ Watch: Introducing the five pillars of incident response (4:15)

Pradhan meticulously breaks down his five pillars of incident response, providing actionable insights for each.

1. Detection: Normalizing the Chaos

The first pillar, Detection, focuses on consolidating disparate alert sources. In early-stage companies, alerts can originate from various systems: a Security Information and Event Management (SIEM), a ticketing system like Jira, customer support channels, or even direct emails from the CEO. This "insane" volume and variety of intake points overwhelm a single security professional. The solution is to establish a single pane of glass—a unified platform or interface where all alerts are funneled and normalized. While achieving identical formatting for every alert is a long-term goal, the immediate objective is to ensure all incident notifications arrive in one centralized location, reducing the cognitive load and panic during an actual event.

2. Command Structure: Defining Roles

The Command Structure pillar addresses the often-ambiguous leadership in incident scenarios. Pradhan notes that "everyone thinks they are not the incident commander and then at the same time everyone thinks that they are the incident commander." To counteract this, clear roles and responsibilities are essential. Key roles typically include:

• Incident Commander (IC): The director of the "symphony," responsible for overall direction and decision-making.
• Technical Lead (Mechanic): The individual actively performing technical response actions to stop the threat.
• Communications (Comms) Lead: Responsible for internal and external stakeholder updates.

For small teams, one person may initially fill all three roles. Regardless, the golden rule is that "every incident must have one single commander" to ensure clear authority and prevent conflicting directives.

3. Coordination: Automating the War Room

Even with clear roles, effective Coordination can be undermined by fragmented communication (Slack DMs, email threads). Pradhan emphasizes the need to automate the incident war room. Upon receiving a high-fidelity alert, the system should automatically:

• Create a dedicated Slack channel (or similar chat platform).
• Spin up a Jira ticket (or equivalent incident management record).
• Page the relevant on-call engineer from the security team and potentially sister teams.

The crucial aspect here is that all updates and conversations must occur within this single, automated channel. This ensures that anyone joining the incident response, especially if the initial commander is unavailable, has immediate access to the full context, preventing information silos and reducing the "improvisation tax."

4. Execution: Containment with Guardrails

The Execution pillar focuses on taking decisive action. Pradhan's north star is to achieve containment of the attacker even before the investigation starts. This means having the capability to "press the red button" to disable accounts, quarantine infected endpoints, or revoke compromised keys. However, he cautions against blind automation, advocating for guardrails.

The proposed logic is an if condition:

• IF the asset is not protected (e.g., non-critical server, standard employee account), then move towards auto-containment.
• IF the asset is protected (e.g., critical production database, executive account), then ensure a human in the loop for decision-making.

This requires significant context enrichment. Pradhan illustrates this with an example: Alice suspiciously logs in from a Russian IP, and threat intelligence indicates ransomware. However, the host is a production database owned by the infrastructure team. Without enrichment, an automated response might shut down the database. With context, the human in the loop would first consult the infrastructure team to understand the database's criticality and potential downstream impact, leading to a more informed decision. Automated enrichment should pull in details like user roles, device ownership, IP geotagging, and threat intelligence feeds.

5. Governance & Preservation: Beyond Technical Response

The final pillar, Governance & Preservation, is often overlooked but is critical for the long-term health of the company. Pradhan states, "technical response saves the team but compliance and governance response saves the company." This involves:

• Breach Notifications: Understanding regulatory requirements and timelines for informing affected parties and authorities (e.g., GDPR, CCPA). This necessitates engaging legal and compliance experts early.
• Forensic Preservation: Capturing snapshots or images of affected systems before and after any execution actions. This provides crucial evidence for post-incident analysis, audits, and potential legal proceedings, ensuring the integrity and soundness of the forensic data.

Maturity Model and AI Integration

Pradhan outlines an incident response maturity model:

1. Manual Response: The starting point, where the pipeline is built but actions are manual.
2. Automated Triage & Enrichment: Structuring and automating initial data collection and context gathering.
3. Automated Containment: Implementing the "red button" with guardrails for non-critical assets.
4. Proactive Threat Hunting: The ultimate goal, where analysts shift from reactive operators to strategic decision-makers.

He then addresses AI's role, identifying "sweet spots" where it excels:

• Alert Normalization and Context Enrichment: AI can rapidly process disparate data sources, pull in relevant information (device OS, user role, threat intel), and present it in a unified format.
• Investigation Assistant: AI can answer questions about past incidents, correlate Indicators of Compromise (IoCs), and help determine persistence.
• Documentation and Feedback Loop: AI can assist in generating post-mortem reports and identifying areas for process improvement.

However, Pradhan strongly cautions against blindly trusting AI in critical areas:

• Fully Autonomous Containment: Too risky without robust safety nets for protected assets.
• Legal and Compliance Decisions: These are highly nuanced and region-dependent, requiring human legal expertise.
• Evidence Handling: Requires strict, auditable procedures that AI alone cannot guarantee.

The overarching message is that AI should be a tool that augments human capabilities, buying time and enhancing decision support, rather than fully replacing human judgment in high-stakes scenarios.

Demo / Proof of Concept

▶ Watch: Pillar 1: Normalizing chaos and alert intake (5:00)

While no live demonstration of a tool or platform was presented, Geet Pradhan effectively illustrated the power of an automated incident response pipeline through a conceptual "50-minute incident" comparison. This served as a compelling proof of concept for the time-saving benefits of his proposed framework.

The Manual Response Scenario:

Pradhan depicted a typical manual response to a suspicious login alert:

• 0-5 minutes: Alert received, on-call engineer wakes up, takes time to review.
• 5-10 minutes: Triage begins – manual lookup of host, IP, user context (e.g., in Workday or Octa).
• 10-15 minutes: Alert confirmed, ticket opened, initial team communication.
• 15-20 minutes: Further manual log analysis, baseline comparison, threat intelligence lookups.
• 20-30 minutes: Decision to block IP and disable user (still considered aggressive for manual).
• 30-50 minutes: Incident commander (e.g., application owner) finally logs on, taking over the rest of the response.

In this scenario, initial containment takes at least 15-20 minutes, and critical decision-makers are often delayed, contributing significantly to the "improvisation tax." Every step represents a manual check, losing precious time.

The Automated Pipeline Scenario:

In contrast, Pradhan outlined how an automated pipeline drastically reduces response time:

• 0-1 minute: Alert received.
• 1-2 minutes: Automated processes kick in:
• Jira ticket automatically created.
• Dedicated Slack channel automatically spun up.
• On-call engineer and relevant sister teams automatically paged.
• All initial context and enrichment (IP, identity, user role, threat intelligence, asset ownership) automatically added to the Jira ticket and Slack channel.
• 2-5 minutes: Based on predefined guardrails (e.g., "if not a protected account"), the user is automatically disabled.
• 5-7 minutes: The host is automatically isolated, and forensic snapshots are captured (before and after action).

By the time the human on-call engineer even gets to their desk, the initial containment and critical context gathering are largely complete. The engineer's role shifts from reactive data collection to reviewing the scope and making higher-level decisions. This "pipeline buys time," shifting the defender's methodology from reacting to proactively responding, significantly mitigating risk in the crucial early minutes of an attack.

Defensive Implications

▶ Watch: Pillar 2: Defining incident command structure and roles (6:00)

The implications of Pradhan's incident response pipeline for defenders are profound, particularly for those operating in resource-constrained environments. The core message is clear: time is the most critical asset in incident response, and strategic automation is the primary means to acquire it.

Defenders should prioritize "plumbing the pipes" by ensuring robust telemetry and normalizing all alert sources into a single, digestible format. This foundational step eliminates the chaos of disparate systems and provides a unified view, crucial for rapid assessment. Tools like a SIEM or even a well-configured ticketing system can serve as the central hub.

Secondly, context is paramount, and its enrichment should be automated wherever possible. Instead of relying on manual lookups, automated processes should pull in critical information about users, devices, IPs, and threat intelligence before a human even reviews the alert. This proactive enrichment empowers responders with immediate, actionable intelligence, enabling faster, more informed decisions and reducing the "improvisation tax."

Thirdly, the focus must be on containment, containment, containment. Defenders should strive to contain threats as early as possible, ideally before extensive investigation. This involves implementing auto-containment mechanisms for non-critical assets, coupled with robust guardrails and a human-in-the-loop for critical systems. The ability to "press the red button" quickly, while ensuring safety, is a game-changer. Pradhan explicitly states, "Containment beats documentation," emphasizing that stopping the attack takes precedence over administrative tasks in the initial moments.

Furthermore, defenders must integrate governance and forensic preservation into their pipeline from the outset. Understanding breach notification timelines and having procedures for capturing forensic snapshots before and after response actions are vital for compliance, legal defense, and post-incident analysis.

Finally, while AI offers significant advantages in areas like alert correlation and investigative assistance, defenders must approach its deployment with caution, particularly for fully autonomous containment or legal decisions. A human oversight layer remains indispensable for high-stakes actions. For those starting from scratch, Pradhan points to free resources and open-source tools that can help build a demo environment and gain practical experience, emphasizing that an effective pipeline doesn't necessarily require significant financial investment. The continuous feedback loop, through post-mortems and lessons learned, is also a non-negotiable for maturing the IR program.

Key Takeaways

• Time is the Ultimate Currency: In cyber incident response, the ability to save minutes and hours by shifting from reactive panic to structured response is paramount. Automation is the key mechanism to "buy time."
• The Five Pillars Framework: A robust incident response pipeline is built upon five interconnected pillars: normalized Detection, clear Command Structure, automated Coordination, decisive Execution (with guardrails), and diligent Governance & Preservation.
• Automate the War Room and Enrichment: Centralize all incident communications in an automatically created "war room" (e.g., Slack channel, Jira ticket) and automate the enrichment of alerts with critical context (user, device, IP, threat intel) before human intervention.
• Prioritize Containment with Guardrails: Establish the north star of containing the attacker even before investigation. Implement auto-containment for non-critical assets, but always include a human-in-the-loop for protected or critical systems to prevent unintended downstream impact.
• Strategic AI Integration: Leverage AI for "sweet spots" like alert normalization, context generation, and investigative assistance to augment human capabilities. However, exercise caution and maintain human oversight for fully autonomous containment, legal decisions, and evidence handling.
• Continuous Improvement through Feedback: Post-mortems and "lessons learned" sessions are crucial for identifying what went right and wrong, feeding back into the pipeline to continuously improve detection rules, processes, and overall response efficacy.

About the Speaker(s)

Geet Pradhan is a security engineer with extensive experience in building incident response programs. He describes himself as the "concerto conductor of the beautiful symphony of incident response," highlighting his passion for orchestrating effective security operations. Pradhan specializes in developing security capabilities "from zero to one," particularly for organizations with limited resources and lean teams. He has previously spoken at BSides SF (in 2025, a humorous anachronism given the conference year) on the topic of detection pipelines, which focused on identifying high-fidelity signals from noise. His current talk extends this work, emphasizing the creation of an "actionable shield" to respond to those signals.