Product Security Incident Response at a Fortune 500 SaaS

Product Security Incident Response at a Fortune 500 SaaS

Speakers: Garrett, Product Security Lead, ServiceNow

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=MJffUd1kzF0

Overview

This talk, presented by Garrett at VulnCon, delves into the intricate world of Product Security Incident Response Teams (PSIRTs) within a large Software-as-a-Service (SaaS) organization, specifically a Fortune 500 company like ServiceNow. Garrett, who was instrumental in establishing ServiceNow's PSIRT, provides a candid and detailed account of the unique challenges and opportunities that arise when managing product security incidents in a cloud-hosted environment, contrasting it with traditional on-premise software vendors.

The presentation highlights that while SaaS offerings simplify many aspects for customers, they introduce a distinct set of security complexities for vendors. These include rapid attack surface discovery, heightened stakes due to internet connectivity, nuanced disclosure policies, and the delicate balance between security posture and customer operational stability. Garrett's insights are crucial for security professionals navigating the complexities of modern cloud security, emphasizing the need for adaptable strategies that account for shared responsibility, customer diversity, and the ever-evolving threat landscape.

This article aims to dissect Garrett's experience, offering a comprehensive look at the specific issues encountered and the strategies employed within a high-stakes SaaS environment. It underscores the critical importance of a robust and agile PSIRT that can respond effectively to vulnerabilities while maintaining customer trust and business continuity.

Background

▶ Watch: Speaker introduction and starting ServiceNow's PERT (0:00)

Garrett's journey into product security is rooted in a background as a web application developer, which instilled a deep appreciation for security early on. This led him into security research and later, an role as an application security educator, working directly with developers to embed security best practices into the software development lifecycle. Approximately two and a half years prior to this talk, Garrett was tasked with coaching and establishing ServiceNow's Product Security Incident Response Team from the ground up.

ServiceNow, a Fortune 500 SaaS company, already operated a robust bug bounty program and a responsible disclosure initiative. However, these efforts lacked a centralized, official "house" and dedicated team to manage product security incidents comprehensively. Garrett's mandate was to formalize this function, growing the team from its inception to 12 engineers spanning three continents. The team is now a recognized CNA (CVE Numbering Authority) and a member of first.org, underscoring its commitment to industry best practices and vulnerability coordination. Garrett's extensive experience, which includes prior PSIRT roles since 2015 at companies like Forcepoint (also a first.org member), Octa (where he gained 18 months of "refresher" experience in prevention and education), Invincia, and various government contractors, provides a multifaceted perspective on the evolution from on-premise to cloud-hosted security models.

The core premise of Garrett's talk is that PSIRTs operating within SaaS environments face distinct challenges and opportunities compared to those at vendors primarily focused on on-premise software. While SaaS infrastructure, by its nature, is often internet-connected and easily discoverable by attackers, it also offers unique advantages. These include enhanced telemetry and detection signals that enable risk-based decision-making, allowing PSIRTs to prioritize and expedite fixes based on real-time threat intelligence and potential business disruption. This dynamic environment necessitates a different approach to incident response, one that is agile, customer-aware, and highly technical.

Key Findings

▶ Watch: Key risk factor: speed of attack surface discovery (4:10)

Garrett's presentation meticulously outlines the dual nature of SaaS PSIRT, dedicating approximately 70% of his discussion to challenges and 30% to opportunities. The central finding is that while SaaS offers inherent advantages in observability and rapid deployment, it simultaneously introduces a host of complexities that demand sophisticated incident response strategies.

Primary Challenges Identified:

• Rapid Attack Surface Discovery: SaaS platforms are inherently internet-connected and easily discoverable, providing convenient access for both legitimate researchers and malicious actors. The use of subdomains can facilitate rapid enumeration of other customer instances.
• High Stakes of Vulnerabilities: Due to internet exposure, many vulnerabilities in SaaS environments, when assessed using CVSS (Common Vulnerability Scoring System), frequently register the highest "Network" attack vector, leading to inherently higher severity scores. This is compounded by reduced defense-in-depth compared to isolated on-premise deployments.
• Misunderstood Shared Responsibility Model: Customers often misinterpret their role in security, realizing too late during a crisis that they have a part to play, leading to unexpected demands on the vendor.
• Limited Vendor Visibility: SaaS vendors may intentionally lack visibility into customer request/response data (for privacy, liability, or contractual reasons), making it difficult to advise on specific breach occurrences or data leaks.
• WAF and Mitigation Limitations: Generic Web Application Firewall (WAF) rules or rate-limiting strategies can inadvertently break highly customizable software or disrupt legitimate automated customer jobs, preventing universal application.
• Complex Disclosure Decisions: Determining when and how to disclose vulnerabilities (e.g., issuing a CVE) is fraught with dilemmas. Should a CVE be issued if no customer action is required? How soon after a patch should full details be published, balancing researcher interest with customer testing timelines (which can range from 30 days to 6-12 months)? Embargoes become incredibly complex with thousands of diverse customers.
• Customer Intent and Breaking Changes: It's challenging to ascertain if customers intentionally configured their systems in a vulnerable manner or if legacy setups/accidental builds on insecure foundations will break with underlying platform changes. Garrett describes a "multi-ring model" for breaking changes, impacting the vendor's platform, vendor-built applications, and customer-written code.
• Enablement Paradox: Vendors are pressured to ship secure products but must also offer options for backward compatibility or lesser security configurations to meet diverse business needs. If a customer chooses a less secure option and is breached, the vendor's brand still suffers.
• Communication Failures: Outdated customer contact information, lack of backup contacts, and organizational silos (e.g., security team vs. maintenance team) can impede critical communications during incidents. The ecosystem of implementation partners can further complicate message delivery to end customers.
• "Living Off the Land" Abuse: Powerful, legitimate product features, designed to solve complex customer problems, can be abused (maliciously or accidentally) for computationally intensive tasks or to interact excessively with external services, burdening the vendor.
• Broad Infrastructure Responsibility: SaaS PSIRTs are responsible not only for their product but also for the entire underlying infrastructure, including load balancers, backend services (AI, identity), and storage. Furthermore, even predominantly hosted vendors must consider the security implications of any on-premise software they distribute, as it can be reversed by researchers.
• "Steering" and Rollback Restrictions: Vendors may block customers from rolling back to vulnerable versions or installing older, insecure app versions. This "opinionated" stance prioritizes security but can cause significant operational disruption for customers who require extensive testing time before adopting new versions.

Key Opportunities Identified:

• Honeypot Potential: Demo or non-production instances, if scanned by attackers, can inadvertently function as honeypots, providing valuable threat intelligence.
• Enhanced Observability: The ability to monitor exploit attempts, scanning activity, version adoption rates, specific configuration usage, and application data volume allows PSIRTs to make risk-based decisions, accelerating remediation for high-impact, actively exploited vulnerabilities beyond standard SLAs (e.g., fixing criticals in 15 days).
• Force Changes: In critical situations, vendors can force changes onto customer instances or urgently mandate customer action, rapidly mitigating widespread threats. However, this is highly disruptive and can inadvertently set a precedent that undermines the shared responsibility model.

Garrett also highlights a "bonus opportunity": PSIRTs gain a unique, holistic perspective on systemic weaknesses, observing all areas of improvement—technical, operational, human, and process-related—that lead to vulnerabilities reaching their team. This vantage point is invaluable for driving upstream security enhancements.

Technical Deep Dive

▶ Watch: High stakes: internet connectivity and CVSS network vector (5:40)

The technical intricacies of a SaaS PSIRT, as described by Garrett, revolve around managing a sprawling, internet-exposed attack surface with diverse customer needs and varying levels of security maturity.

The speed of attack surface discovery is a critical technical concern. Unlike on-premise software requiring manual installation and configuration, SaaS applications are readily accessible. Researchers and attackers can quickly enumerate potential targets, especially when customers are hosted on subdomains (e.g., customer.company.com). The barrier to entry for finding basic web vulnerabilities is low, often requiring only a browser or free tools, without the need for complex malware reversing or chipset manipulation. This rapid discovery pace means PSIRTs must be exceptionally agile.

The high stakes are technically quantified by the CVSS Attack Vector: Network, which consistently yields the highest score component for internet-facing vulnerabilities. This implies that many SaaS vulnerabilities start with a high base severity, demanding immediate attention. Furthermore, the hosted nature often means less defense-in-depth at the customer's perimeter, as the vendor's infrastructure is directly exposed. Ingress and egress requirements, often dictated by customer preferences, can further limit a vendor's ability to implement universal network-level controls.

The shared responsibility model, while conceptually sound, often breaks down in practice. Technically, vendors may design their systems to not have visibility into customer request/response data, citing liability or contractual obligations. This means that when a customer asks "Was I breached?", the vendor may legitimately not be in a position to answer, leading to frustration and potential misunderstandings. Implementing a universal Web Application Firewall (WAF) is technically challenging for highly customizable SaaS platforms, as rules that protect one customer might inadvertently break the specific configurations or integrations of another. Similarly, rate limiting, a common defensive control, can disrupt legitimate automated jobs run by 90-95% of customers, impacting critical business processes.

Vulnerability disclosure is a technically and strategically complex area. Garrett discusses the debate around issuing CVEs when no customer action is required. While some leading-edge companies disclose all critical and high severity issues, others believe it merely creates "noise." The timing of disclosure is also crucial: should a patch be released with generic security notes, allowing customers 30 days (or even 6-12 months for some) to test before full CVE details are published? Or should CVEs be issued immediately, as some management tools only trigger action based on formal CVEs, leading to "holdouts" if delayed? The sheer scale of "thousands of customers" makes embargoes incredibly difficult, if not impossible, to manage effectively.

Customer intent and the impact of breaking changes present significant technical hurdles. Customers may unknowingly build on top of insecure configurations from years past. When the vendor changes underlying platform behavior for security reasons, it can break not only vendor-provided applications but also customer-developed code. Garrett's "multi-ring model" illustrates this: the vendor must account for changes to the core platform, vendor-supplied applications, and customer-written code that relies on the platform. Assessing customer modifications to files (e.g., when checksums no longer match) makes it risky for vendors to automatically apply fixes, as they might inadvertently modify customer-owned code.

Enablement involves the technical trade-off between shipping secure defaults and providing options for less secure, legacy configurations (e.g., for older ciphers or lack of 2FA). While the vendor offers these options, the brand still takes a hit if a customer, having opted for a less secure path, experiences a breach.

Communication failures are not strictly technical but have technical implications. Outdated contact information means automated security notifications fail. The disconnect between security and maintenance teams within customer organizations can delay patch adoption. Furthermore, the broader ecosystem of implementation partners can act as a single point of failure for critical security messages reaching the end customer.

Living off the land describes the abuse of powerful, legitimate product features. Technically, these features, designed for complex problem-solving, can be exploited for computationally intensive tasks or to interact with external services (e.g., APIs), leading to resource exhaustion or external abuse reports (e.g., "your infrastructure is abusing mine"). This can be due to malicious intent or simply poorly written, non-optimized customer code (e.g., infinite loops).

SaaS PSIRTs are also technically responsible for the entire stack. This includes not just the product application code but also the underlying infrastructure components like load balancers, backend AI services, identity management systems, and storage. For vendors with an on-premise business model, even if it's a small percentage of their customer base, the distribution of software (obfuscated or not) means it can be reversed by researchers, negating any assumption that internal mitigations will remain unseen.

Finally, steering involves the technical control of blocking customers from rolling back to vulnerable versions or installing older, insecure applications from an app store. While this improves security, it creates a conflict with customer testing cycles (e.g., 30 days for sub-production, 6-12 months for production) versus researcher disclosure timelines (typically 90 days). Forcing changes is a powerful technical capability, but it is highly disruptive, can lead to service outages (impacting availability, a core security tenet), potentially violating SLAs, and may set a negative precedent where customers expect the vendor to always fix issues for them.

Demo / Proof of Concept

▶ Watch: Shared responsibility model for cloud security (6:30)

The talk did not include a specific demonstration or proof of concept. Garrett's presentation was primarily a discussion of the challenges and opportunities in Product Security Incident Response within a SaaS environment, drawing from his extensive experience at ServiceNow and other technology companies.

Defensive Implications

▶ Watch: Challenge: vendor visibility into customer instances (7:40)

The insights shared by Garrett carry significant defensive implications for both SaaS vendors and their customers.

For SaaS Vendors:

• Proactive Attack Surface Management: Given the rapid discoverability of SaaS infrastructure, vendors must maintain a continuous and comprehensive understanding of their public-facing assets. This includes diligent inventory of subdomains, understanding their exposure, and conducting regular external attack surface assessments.
• Risk-Based Prioritization: Leverage observability and telemetry to identify active exploit attempts or widespread scanning activities. This data should inform a dynamic prioritization model that can accelerate remediation beyond standard SLAs for critical, actively exploited vulnerabilities.
• Refined Disclosure Policies: Develop nuanced vulnerability disclosure policies that consider the SaaS context. This involves clear guidelines on when to issue CVEs (even if no customer action is required), the timing of full detail publication, and strategies for managing complex embargoes across a large customer base. Transparency, while valuable, must be balanced with avoiding customer alert fatigue and providing actionable information.
• Enhanced Customer Communication: Establish robust channels for security communications, ensuring contact information is always current and that backup contacts are available. Recognize the potential for organizational silos within customer environments and tailor messaging to reach both security and operational teams effectively.
• Educate on Shared Responsibility: Continuously educate customers on their role within the shared responsibility model. Clearly articulate what the vendor is responsible for and where customer action is required, especially concerning data visibility, configuration choices, and patch adoption.
• Intelligent Mitigation Strategies: Avoid "one-size-fits-all" security controls like universal WAF rules or aggressive rate limiting, as they can break highly customizable software. Instead, develop flexible, context-aware mitigations that can be applied granularly or adapted to specific customer needs.
• Secure by Default, but with Awareness: While shipping products that are "secure by default" is paramount, vendors must be acutely aware of the implications of offering less secure, backward-compatible options. Understand the brand risk if customers choose these options and experience a breach, and consider strategies to guide customers towards more secure configurations.
• Address "Living Off the Land" Risks: Design powerful features with abuse in mind. Implement controls (e.g., resource quotas, anomaly detection) to prevent or mitigate the impact of computationally intensive abuse or excessive interaction with external services, whether malicious or accidental.
• Holistic Security: Recognize that PSIRT responsibility extends beyond product code to the entire underlying infrastructure. Invest in security for load balancers, backend services, identity, and storage, and account for the security implications of any distributed on-premise software.
• Strategic "Steering": While blocking rollbacks to vulnerable versions can enhance overall security, this must be balanced with customer operational needs and testing timelines. Clearly communicate the rationale and potential impacts of such policies to avoid unexpected disruption.
• Honeypot Utilization: If demo or non-production environments are scanned, analyze the collected data for threat intelligence. Ensure these environments are patched as rigorously as production instances to avoid negative brand perception if compromised.

For SaaS Customers:

• Understand Shared Responsibility: Actively engage with your SaaS vendors to understand the precise boundaries of the shared responsibility model. Know what security responsibilities fall to you (e.g., configuration, user access, data handling) versus the vendor.
• Maintain Current Contacts: Ensure your vendor has up-to-date security contact information, including backup contacts, to receive critical security advisories promptly.
• Proactive Patch Adoption: Don't solely rely on CVEs to trigger action. Pay attention to all vendor security communications and prepare to test and adopt patches quickly, balancing your operational stability with the need for timely remediation.
• Thorough Testing: While vendor timelines may be tight, conduct thorough testing of updates in non-production environments to identify any breaking changes before deploying to production.
• Secure Configurations: Avoid using legacy or less secure configuration options unless absolutely necessary, and understand the associated risks. Regularly review your configurations to ensure they align with best practices.
• Monitor Your Usage: Be aware that powerful features of your SaaS platform can be abused. Monitor your own automated jobs and custom code for inefficiencies or unintended resource consumption that could lead to "living off the land" issues.

Key Takeaways

• SaaS PSIRT is a Distinct Discipline: Operating a PSIRT in a Fortune 500 SaaS environment presents unique challenges and opportunities that fundamentally differ from traditional on-premise software vendors, necessitating specialized strategies.
• Rapid Attack Surface Discovery and High Stakes: The internet-connected nature of SaaS platforms leads to rapid attack surface discovery and often results in higher CVSS scores (due to the "Network" attack vector), demanding an agile and highly responsive incident response.
• Complex Disclosure and Communication: Deciding when and how to disclose vulnerabilities to thousands of diverse customers, manage embargoes, and communicate effectively through potentially siloed customer organizations is a significant challenge.
• Leveraging Observability for Risk-Based Decisions: SaaS vendors have the opportunity to utilize telemetry and exploit attempt data to prioritize and accelerate remediation efforts for the most critical and actively exploited vulnerabilities, going beyond standard SLAs.
• Balancing Security with Customer Operations: PSIRTs must navigate the delicate balance between enforcing strong security postures (e.g., blocking rollbacks to vulnerable versions) and ensuring customer operational stability, recognizing that disruptive "force changes" can have negative long-term implications for shared responsibility.
• Holistic Responsibility: SaaS PSIRTs are accountable not only for the product application code but also for the entire underlying infrastructure, including load balancers, backend services, and storage, necessitating a broad security mandate.

About the Speaker(s)

The speaker, Garrett, is a seasoned security professional who has been involved with Product Security Incident Response Teams (PSIRTs) since 2015. He began his career as a web application developer but quickly gravitated towards security research and later served as an application security educator. Approximately two and a half years prior to this talk, Garrett was coached to establish and lead ServiceNow's PSIRT, growing the team to 12 engineers across three continents.

Garrett's extensive experience includes prior roles at Forcepoint, where he was involved with a CNA and first.org, and Octa, where he gained valuable experience in prevention and education within the security lifecycle. He also worked with Invincia and various government contractors. Currently, in addition to his role at ServiceNow, Garrett is pursuing an MBA, approximately halfway through his studies. In his limited free time, he previously dedicated six to seven years to search and rescue operations. His career progression from on-premise to hosted environments provides him with a unique perspective on the evolving landscape of product security.