From NIST to FIRST: How GitHub’s Product Security Response Organization Transitioned

From NIST to FIRST: How GitHub’s Product Security Response Organization Transitioned

Speakers: Sarah Clemens, Peer Manager, Bug Bounty and Research Teams, GitHub; Jeff, Vulnerability Management Program, GitHub

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=2HKlNgHmp7k

Overview

In this VulnCon talk, GitHub’s Sarah Clemens and Jeff detail their organization's methodical journey in evolving its security incident response capabilities. The presentation focuses on GitHub’s strategic transition from a monolithic incident response team to a specialized Product Security Incident Response Team (PSIRT), emphasizing how this transformation was guided and benchmarked against industry-leading frameworks. The speakers illuminate the critical decision to separate product-focused security from broader corporate incident response, a move driven by the need for enhanced prioritization and specialized expertise in a rapidly growing company.

Clemens and Jeff articulate how GitHub leveraged established security frameworks—specifically NIST 800-61, the SIM3 (Security Incident Management Maturity Model), and the FIRST (Forum of Incident Response and Security Teams) PSIRT Services Framework—to structure, mature, and sustain their program. The talk provides a high-level yet insightful look into the practical application of these frameworks, demonstrating how they informed the development of robust processes, stakeholder engagement strategies, and continuous training initiatives. This strategic alignment not only fortified GitHub's ability to respond to product vulnerabilities but also streamlined internal operations, including audit preparedness, by adopting a shared industry language.

The insights shared are particularly relevant for organizations grappling with the complexities of scaling security operations, managing diverse stakeholder expectations, and integrating external security research efforts like bug bounty programs into their core incident response workflows. GitHub's experience underscores the importance of a well-defined, adaptable, and continuously maturing PSIRT, especially in the context of a company that builds and secures its own products using its platform. The speakers also touch upon contemporary challenges, such as managing the influx of AI-generated vulnerability reports, highlighting the ongoing need for both technical and communicative agility.

Background

▶ Watch: Introduction and talk agenda (0:00)

Before the advent of a dedicated PSIRT, GitHub operated with a single, centralized incident response (IR) team. This original CERT (Computer Emergency Response Team) was tasked with handling both corporate incident response (CSIRT) and product incident response, a common structure in many organizations during their formative stages. The foundational principles for this early IR program were derived from the controls outlined in NIST Special Publication 800-61, a widely recognized guideline for computer security incident handling.

NIST 800-61 provided a robust starting point, establishing core fundamentals such as a clear Process and Plan (encompassing the overarching incident response plan and subsequent playbooks), comprehensive Communication protocols for a broad spectrum of internal and external stakeholders (including engineers, product owners, legal teams, users, and even law enforcement), a well-defined Scope detailing what types of incidents were covered and when, and dedicated Staffing of trained responders. These NIST-derived fundamentals were crucial for establishing the initial tone and culture of GitHub's IR program and securing vital leadership buy-in, which is paramount during crisis situations.

However, as GitHub experienced significant growth and expansion, the limitations of a combined CERT became apparent. The primary challenge was prioritization. It became increasingly difficult to allocate resources effectively between corporate-level incidents (e.g., internal system breaches) and product-level vulnerabilities (e.g., security flaws in GitHub’s core offerings). This often led to scenarios where resource allocation felt like a "conflict of interest" during peak crisis times, hindering optimal response for both domains. Recognizing this bottleneck, GitHub made the strategic decision to split the single CERT into two distinct entities: a CSIRT focused on corporate incident response and a PSIRT dedicated to product security incident response. This separation allowed for specialized focus, tailored processes, and optimized resource allocation for each critical area.

With the establishment of the new PSIRT, GitHub sought to deepen its capabilities and mature its program further. This led them to the FIRST (Forum of Incident Response and Security Teams), a global leader in incident response. To become a member of FIRST, organizations must meet a baseline level of maturity, which is often assessed using the SIM3 (Security Incident Management Maturity Model) framework. While SIM3 is primarily geared towards CSIRT maturity, it provided GitHub with a valuable, objective scale (0-4) to assess and improve its incident response capabilities, even as they were building a product-specific team. Achieving this baseline maturity was a significant undertaking, requiring a year of dedicated effort, persistent coordination, and extensive documentation, laying the groundwork for a more specialized and effective product security response.

Key Findings

▶ Watch: Initial Program Foundation: NIST 800-61 (3:00)

GitHub's journey from a unified CERT to a distinct PSIRT, guided by industry frameworks, yielded several key findings and contributions that offer valuable lessons for other organizations:

First, the strategic decision to split the original CERT into separate CSIRT and PSIRT functions was crucial for addressing prioritization conflicts and enabling specialized focus. This separation allowed GitHub to allocate resources more effectively and develop tailored incident response processes for both corporate and product-specific security challenges, enhancing overall organizational resilience.

Second, the systematic adoption and integration of security frameworks—NIST 800-61, SIM3, and the FIRST PSIRT Services Framework—proved instrumental in structuring, maturing, and benchmarking GitHub's security response capabilities. NIST provided the essential organizational foundations, SIM3 offered a quantifiable maturity model for achieving FIRST membership, and the FIRST PSIRT Services Framework specifically catered to the nuances of product security, providing a direct roadmap for PSIRT development.

Third, GitHub demonstrated the profound importance of strong organizational foundations for any IR program. The NIST-derived principles of clear processes, robust communication channels (like their internal stakeholder rolodex and service catalog), well-defined scope, and dedicated, trained staff were identified as critical for setting the program's culture and ensuring leadership buy-in, especially during crises.

Fourth, the talk highlighted the successful achievement of FIRST membership baseline maturity through a year-long dedicated effort using the SIM3 framework. This involved a rigorous self-assessment and improvement process across operational organization, human aspects (resiliency, training), tools, and processes, showcasing GitHub's commitment to industry best practices.

Fifth, the FIRST PSIRT Services Framework was identified as a powerful tool for maturing a product-specific IR team, particularly due to its close alignment with GitHub's bug bounty program and the entire vulnerability lifecycle. This framework enabled GitHub to build a more holistic, first-party vulnerability management program, fostering tighter partnerships with engineering teams.

Finally, GitHub's experience underscored the significance of innovative researcher engagement and continuous training. By gamifying bug hunting (e.g., through a swag shop tied to valid vulnerability submissions), conducting public spotlight interviews with researchers, and maintaining a public presence at conferences, GitHub actively cultivated its external security research community. Internally, rigorous first responder training, extensive documentation, shadowing, cross-training, and knowledge dumps were emphasized as vital for maintaining a highly skilled and adaptable PSIRT. The talk also acknowledged the emerging challenge of managing AI-generated "noise" in bug bounty submissions, indicating a proactive approach through researcher education and internal communication alignment.

Technical Deep Dive

▶ Watch: Why GitHub Split CERT and PERT (6:00)

GitHub's evolution of its security incident response program is a testament to the practical application of industry-standard frameworks. The journey began with NIST Special Publication 800-61, which provided the initial bedrock for their combined CERT. The talk highlighted four core NIST-derived fundamentals:

1. Process and Plan: This isn't just a playbook, but the foundational Incident Response Plan upon which all specific playbooks are built. It defines the overarching strategy and workflow for incident handling, ensuring consistency and reliability in response, even if individual responders internalize the process over time.
2. Communication: This encompasses a vast array of stakeholders, both internal (engineers, product owners, legal) and external (users, law enforcement, international governing bodies). GitHub maintains an internal stakeholder rolodex and a service catalog to quickly identify product owners, streamlining communication during active incidents.
3. Scope: Defining what vulnerabilities and incidents the team covers, and under what circumstances, is critical. For a growing company like GitHub, this is a "moving target" as products and resources evolve, making the continuous refinement of scope a significant challenge.
4. Staffing: This involves establishing dedicated and trained responders. The initial NIST foundation mandated core IR teams with clear responsibilities and ongoing training to maintain proficiency.

As GitHub grew, the decision to separate the corporate (CSIRT) and product (PSIRT) functions led them to seek further maturity, particularly for the new PSIRT. This brought them to FIRST and its associated SIM3 (Security Incident Management Maturity Model). SIM3 assesses IR program maturity on a 0-4 scale, primarily geared towards CSIRT operations but serving as a crucial baseline for FIRST membership. GitHub spent approximately a year working towards this baseline.

The speakers provided an example using the "Staffing" dimension of SIM3:

• Level 0: No IR capability.
• Level 1: Dedicated IR individual contributors (ICs).
• Level 2: Minimum dedicated ICs documented in policies and standards.
• Level 3: Minimum dedicated ICs documented, with leadership approval and upholding of the policy.
• Level 4: Yearly reassessment and update of staffing capacity and needs, with an official approval chain.

This systematic approach, requiring "persistence, wide-reaching coordination skills, and furious docs writing capability," allowed GitHub to achieve a maturity level (often "green" on their internal SIM3 assessment chart) that made them confident in their FIRST application. The SIM3 assessment also broke down maturity across Operational Organization, Human Aspect (resiliency, training), Tools (though acknowledged as more CSIRT-focused), and Processes.

The true deep dive into product security came with the FIRST PSIRT Services Framework. This framework is specifically designed for product-focused incident response and significantly informed GitHub's PSIRT maturation. It's structured around:

• Organizational Foundations: At the top, aligning closely with their initial NIST work.
• Vulnerability Lifecycle: The core of the framework, detailing phases of vulnerability management.
• Stakeholder Ecosystem Management: Emphasizing interactions with various internal and external parties.
• Training and Education: Underpinning all aspects of the PSIRT.

Jeff elaborated on the Vulnerability Lifecycle within this framework, highlighting how GitHub's bug bounty program is a "fundamental part" of this cycle, enabling a "more holistic first-party vulnerability management program."

1. Vulnerability Discovery:

• Stakeholder Management: Ensuring proper intake portals for both internal and external submissions. Internal folks use Slack or other mediums, while external researchers use dedicated channels.
• Intake Portals: GitHub utilizes Zendesk and email for general security concerns, and HackerOne for public and private bug bounty programs. These processes are designed to be "consistent, repeatable, and durable."
• Training and Education: Continuous efforts include teaching stakeholders how to use intake portals. For bug bounty, this extends to public-facing initiatives like bug bounty spotlight interviews with researchers, maintaining a public presence at conferences, and gamifying bug hunting. An example given was a swag shop where coupons were only awarded for valid vulnerability submissions, creating a "CTF in itself" incentive. Public disclosures and crediting researchers also fall into this category. The talk mentioned RFC 2350 as a baseline for continuous internal and external education.

1. Triage:

• Automation: Bug bounty engineers track vulnerabilities using automation, which initiates SLAs (Service Level Agreements) for remediation once a vulnerability is confirmed. This includes summarizing findings and artifacts.
• Prioritization: PSIRT analysts lead the prioritization and "commanding" of the incident, tasking action items.
• Product Engineering Involvement: Engineering teams are looped in immediately upon confirmation of a vulnerability, using GitHub issues to secure GitHub.
• Communication: A "blameless" communication culture is critical to accelerate remediation, avoiding accusations and focusing on solutions.
• Training and Education: For bug bounty, this involves training on "tone and channeling the GitHub voice" in communications with researchers. For first responders, extensive documentation, shadowing, cross-training, knowledge dumps, and "reverse shadowing" (where trainees lead and experienced staff observe) are key.

1. Remediation:

• Fix Implementation: Engineering teams push out fixed PRs (Pull Requests). Product owners are involved in any resulting decisions.
• Verification: Bug bounty engineers retest Proof of Concepts (PoCs), look for potential variants (similar vulnerabilities), and discuss rewards based on the source and severity of the finding, leveraging "prior art" and mutation analysis for consistency.
• Training and Education: The GitHub engineering boot camp ensures all engineers understand how to "use GitHub to secure GitHub," including opening and merging PRs. Command training, through shadowing and cross-training, is also vital here.

1. Disclosure:

• Stakeholder Management: The disclosure phase involves a "long" list of stakeholders, each with valid concerns about framing and messaging.
• Decision Making: Disclosure decisions are driven by deadlines and foundational principles like transparency. Options include emails to affected users, CVEs (Common Vulnerabilities and Exposures), blog posts for wider visibility, or simply change logs.
• Leveraging Prior Art: GitHub emphasizes using "prior art" and documented lessons learned to guide disclosure decisions. If a similar vulnerability warranted a blog post in the past, a more severe one would at minimum require the same. This historical documentation helps in making rapid, consistent decisions "when everything looks like it's on fire."
• Training and Education: Consistent communications and language, aligned with the "GitHub voice," are crucial. Documenting lessons learned, though challenging during an incident, is highlighted as incredibly valuable for future decision-making and for audit purposes.

The speakers concluded by noting that aligning with these frameworks and incorporating them into their mission has significant perks, including making "audit season a breeze" due to using shared industry language in their documentation, which their audit team specifically commended.

Demo / Proof of Concept

▶ Watch: Transition to FIRST and SIM3 Framework (7:00)

The talk "From NIST to FIRST: How GitHub’s Product Security Response Organization Transitioned" primarily focused on organizational strategy, framework adoption, and process implementation rather than a technical demonstration. There was no live demo or specific proof of concept of a vulnerability or a tool presented during the session. The content was centered on the architectural and procedural aspects of building and maturing a PSIRT.

Defensive Implications

▶ Watch: SIM3 Staffing Maturity Model Example (8:15)

GitHub's journey offers several critical defensive implications for organizations looking to fortify their security posture and mature their incident response capabilities:

1. Strategic Framework Adoption: Organizations should actively leverage established security frameworks like NIST 800-61, SIM3, and the FIRST PSIRT Services Framework. These frameworks provide a structured approach to building, assessing, and maturing IR programs, ensuring comprehensive coverage and alignment with industry best practices.
2. Specialization of Incident Response: The decision to separate Corporate Incident Response (CSIRT) from Product Security Incident Response (PSIRT) is a vital defensive move for growing companies. This specialization prevents prioritization conflicts, allows for tailored expertise, and optimizes resource allocation, leading to more effective and timely responses to both internal and product-specific threats.
3. Invest in Foundational Elements: A strong foundation, as defined by NIST, is non-negotiable. Defenders must prioritize clear incident response plans, robust communication protocols (including detailed stakeholder maps and service catalogs), well-defined program scope, and dedicated, continuously trained staff. These elements are crucial for establishing program culture, securing leadership buy-in, and ensuring operational efficiency during crises.
4. Integrate Bug Bounty Programs Deeply: Bug bounty programs should not be siloed but integrated as a fundamental component of the PSIRT's vulnerability lifecycle. This fosters a continuous discovery mechanism, leveraging external security researchers to identify vulnerabilities proactively.
5. Proactive Researcher Engagement: Beyond just receiving reports, organizations should actively engage and cultivate their security research community. This includes clear intake processes, gamification (e.g., swag for valid findings), public recognition, and maintaining a presence at conferences. A well-engaged researcher community can lead to higher quality and more timely vulnerability disclosures.
6. Continuous Training and Documentation: Defenders must prioritize ongoing training, cross-training, shadowing, and knowledge sharing within their PSIRT. Equally important is meticulous documentation of processes, decisions, and lessons learned. This institutional knowledge is invaluable for consistent response, onboarding new team members, and streamlining compliance and audit processes.
7. Foster a Blameless Culture: During incident response, a blameless communication culture is essential. Focusing on "why" a vulnerability exists and how to remediate it, rather than assigning blame, accelerates resolution and encourages open communication between security and engineering teams.
8. Prepare for Emerging Threats (e.g., AI-generated Noise): The rise of AI-generated vulnerability reports presents a new challenge in bug bounty programs. Defenders need to develop strategies for managing increased "noise-to-signal ratio," including educating researchers on expectations, refining intake filters, and aligning internal communications on AI product security. This proactive approach helps maintain efficiency and focus on legitimate threats.
9. Leverage Prior Art for Decision Making: Documenting past incidents and disclosure decisions (prior art) is a powerful defensive tool. It enables faster, more consistent decision-making during future incidents, especially when under pressure, ensuring that responses align with established organizational principles and transparency goals.

Key Takeaways

• Frameworks are Foundational: Leveraging industry frameworks like NIST 800-61, SIM3, and the FIRST PSIRT Services Framework is crucial for structuring, maturing, and benchmarking security incident response capabilities.
• Specialization Enhances Response: Separating corporate (CSIRT) and product (PSIRT) incident response functions prevents prioritization conflicts and enables specialized expertise, leading to more efficient and effective security outcomes.
• Bug Bounty Integration is Key: A robust bug bounty program, deeply integrated into the PSIRT's vulnerability lifecycle, is a fundamental asset for continuous vulnerability discovery and holistic first-party vulnerability management.
• Invest in People and Process: Continuous training, cross-training, comprehensive documentation, and a blameless communication culture are vital for building a resilient, adaptable, and highly skilled PSIRT.
• Proactive Researcher Engagement: Actively engaging the external security research community through clear intake, gamification, and public recognition significantly enhances vulnerability discovery and strengthens the security ecosystem.
• Documentation Streamlines Operations: Meticulous documentation of processes, decisions, and lessons learned not only improves incident response consistency but also simplifies compliance, audits, and knowledge transfer within the organization.

About the Speaker(s)

Sarah Clemens is the Peer Manager for the Bug Bounty and Research teams at GitHub. Her professional passion lies in utilizing values alignment frameworks to inform and guide high-risk decisions within security operations. Her expertise centers on program maturity, scalability, and sustainability within the context of incident response.

Jeff works in the vulnerability management program at GitHub, having recently transitioned from the bug bounty program and team. His primary interest and expertise lie in tackling security bugs and vulnerabilities at scale, focusing on efficient and widespread remediation efforts across GitHub's products and platforms.