Getting over the finish line: Loom Security Journey.

Getting over the finish line: Loom Security Journey.

Speakers: Narayan Gowraj, Nishant Jain

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=a3a6MypvDSM

Overview

This talk, "Getting over the finish line: Loom Security Journey," delivered by Narayan Gowraj, Head of Security at Loom, and Nishant Jain, Security Engineer at Loom, provides a comprehensive look into the evolution of a security program within a rapidly growing startup. The presentation chronicles Loom's security efforts from 2021, when Narayan joined as the founding security engineer, through its acquisition by Atlassian at the end of 2023. It delves into the practical challenges and strategic decisions involved in scaling security processes, tools, and automation, fostering cross-team collaboration, and navigating significant security incidents.

The speakers share valuable insights into building a security-conscious culture, managing a bug bounty program, and implementing effective vulnerability management and incident response strategies. A significant portion of the talk is dedicated to a detailed post-mortem of a major security incident in 2022, offering transparent lessons learned. This article aims to distill the technical and strategic wisdom shared, providing a roadmap for other organizations facing similar growth and security challenges.

The journey highlights the importance of adaptability, continuous learning, and a proactive approach to security in a dynamic environment. It underscores that security is not merely a technical function but a deeply integrated aspect of engineering and product development, requiring strong cultural alignment and a commitment to transparency.

Background

▶ Watch: P0 Local File Inclusion to RCE in Interview Process (3:00)

Loom, an asynchronous messaging application designed to improve communication and collaboration through recorded videos, was founded in 2015. By the time Narayan Gowraj joined as the founding security engineer in 2021, the company had already achieved significant milestones, including the launch of its Chrome extension in 2016 and a desktop app in 2018. Prior to Narayan's arrival, Loom had initiated its first pentest and bug bounty program in late 2020, and had achieved SOC 2 Type I compliance.

Upon Narayan's joining, Loom boasted approximately 40 million users and served numerous high-profile customers, necessitating robust protection for its intellectual property. The company was also working towards SOC 2 Type II compliance and had established a procurement strategy. The interview process for the founding security engineer role itself highlighted existing security challenges, including a P0 local file inclusion vulnerability leading to remote code execution, which Narayan had to walk through. This early exposure underscored the immediate need for a dedicated security function to address application DoS issues and the absence of proper rate limits.

The initial phase of Loom's security journey, spanning 2021-2023, focused on establishing foundational security practices while accommodating rapid growth. This period was characterized by the need to balance security enhancements with product velocity, integrate security into engineering workflows, and respond effectively to emerging threats and incidents. The acquisition by Atlassian at the end of 2023 marked a new chapter, but the lessons learned during this formative period remain critical to understanding Loom's security posture.

Key Findings

▶ Watch: Self-DDoS Incident via Chrome Extension WAF Interaction (6:00)

The Loom security journey yielded several key findings and strategic approaches that are highly relevant for organizations building and scaling their security programs:

• Proactive Foundational Security: The initial focus was on addressing immediate, high-impact problems like application DoS and lack of rate limits by deploying AWS WAF as a starter project. This demonstrated a pragmatic approach to building security from the ground up.
• WAF Deployment Nuances: Real-world WAF deployment revealed unexpected challenges, including a self-DDoS incident caused by an overly aggressive Chrome extension and legitimate traffic being blocked due to default user agents (e.g., electron Fetch) being misidentified as bot traffic. This highlighted the importance of understanding customer behavior and thorough testing.
• Cultural Integration and "Fail Forward": Loom fostered a strong security culture emphasizing optimism, transparency, and the "fail forward" principle. This encouraged learning from mistakes without fear of blame, which was crucial for continuous improvement.
• Structured Vulnerability Management: A robust vulnerability management policy was established, featuring CVSS scoring, clear SLAs, leadership reviews, and security-owned compensating controls to manage risk effectively.
• Strategic Security Tooling: Loom adopted a hybrid "buy and build" approach for security tools, prioritizing customization to address context-specific issues rather than generic OWASP Top 10 vulnerabilities. This avoided multi-year deals and ensured tools met specific organizational needs.
• Balanced Product-Security Prioritization: Security efforts were balanced with product feature development using an impact-effort matrix. The "trust but verify" principle was implemented with break-glass scenarios and audited access to sensitive resources, ensuring security without hindering developer velocity.
• Contextual Incident Response: Incident severity (SE0, SE1) was defined based on factors like attack surface, reporting source (internal vs. external), and active exploitation. Security took ownership of developing API rules, SAST checks, and alert fine-tuning to enhance detection and response.
• Major Incident Learnings (CDN Config Change): A critical incident in March 2022, involving a CDN configuration change that led to incorrect user sessions being returned, exposed vulnerabilities related to cookie handling, CDN caching policies (specifically, an AWS managed policy overriding max-age: 0), and the SameSite=None attribute for background requests. This incident necessitated a database and cache rollback, resulting in the loss of 3K Loom videos.
• Post-Incident Remediation: Key actions included setting SameSite to Lax, implementing defense-in-depth measures like low-scope cookies, device binding (based on historical IPs and user agents), and integrating security more deeply into quarterly planning.
• Mature Bug Bounty Program: Loom's bug bounty program, managed on HackerOne, demonstrated that while bounty payouts might decline as a program matures, the quality of reports can remain consistent. Strategies like annual bounty increases, short-term promotions (e.g., 1.5x or 2x bounties), long-term incentives (e.g., $555 for five valuable vulnerabilities), and high efficiency in triage and payment (96% met criteria) were crucial for researcher engagement.
• Custom SAST Rules: Instead of relying solely on generic SAST rules, Loom developed custom rules derived from post-mortems of past incidents and recurring issues. These rules were integrated as inline PR comments to provide developers with context-specific guidance without blocking their workflow.
• Strong Developer-Security Collaboration: A significant finding was the high level of security awareness and collaboration among Loom's developers, who actively cared about security, making it a shared responsibility.
• Robust Infrastructure Security: The close integration of security with the infrastructure team enabled the use of IAC tools (e.g., Terraform) with security linters, a CSPM product for misconfiguration detection, and extensive use of Service Control Policies (SCPs) to enforce guardrails.

Technical Deep Dive

▶ Watch: CDN 'caching optimized' Policy Overrides max-age=0 (15:00)

Loom's security journey began with a pragmatic approach to addressing immediate threats. Given the company's reliance on GraphQL with a single API endpoint, the initial deployment of AWS WAF required careful configuration. Conditions were added to narrow down restrictions to specific GraphQL endpoints, operations, queries, and mutations, moving beyond generic web application firewall rules. The WAF was instrumental not only for DDoS protection but also for bot detection and account creation fraud prevention.

However, the WAF deployment presented unique challenges. One notable incident involved self-DDoS, where a change to the Chrome extension client caused it to aggressively request a specific loom.com endpoint for feature flags on every load. The WAF successfully short-circuited this internal attack. Another learning curve involved blocking legitimate traffic: the desktop app, using an open-source library called electron Fetch, utilized a default user agent. AWS WAF mistakenly identified this as bot traffic, leading to legitimate users, particularly educational users from shared VPNs, being blocked. This underscored the necessity of understanding the customer base and thoroughly testing WAF rules.

Vulnerability management at Loom was structured around a clear policy. It adopted CVSS scoring for standardized severity assessment and established SLAs for remediation. A unique aspect was that compensating controls (e.g., additional monitoring, rate limiting, blast radius reduction) were owned by the security team, ensuring that risks were actively managed even when immediate fixes weren't possible. The team also built a custom threat detection platform to complement commercial tools, focusing on API rules, SAST checks, and alert fine-tuning to improve the signal-to-noise ratio.

A critical technical incident occurred on March 7, 2022, involving a CDN configuration change. The change, intended for static assets like JavaScript and CSS, inadvertently stopped stripping cookies at the CDN level. Previously, cookies were stripped, preventing them from being cached. After the change, cookies were sent to the CDN, which then cached the entire request, including the user's session cookie. This cached response was then served to other users, leading to incorrect user sessions being fetched. The root cause was a combination of factors:

1. The CDN was configured with an AWS managed policy called caching optimized. Despite Cache-Control: max-age=0 being set, this managed policy enforced a minimum cache age of 1 second. This 1-second caching window was sufficient to cause the issue in production but went unnoticed in staging due to low traffic.
2. The cookie's SameSite attribute was set to None. While loom.com and cdn.loom.com are technically "same-site" (sharing the same scheme, top-level domain, and top-level domain plus one), the request for static assets was a background request, not a top-level navigation. If SameSite=Lax had been used, it would have prevented the cookie from being sent with background requests, even to a same-site origin. The decision to use SameSite=None was a "short-term win" to allow Loom videos to be viewed and engaged with on third-party sites (e.g., Google Docs), but it compromised security.

The incident was declared at 11:03 AM, and changes were reverted by 11:10 AM. However, due to caching, support tickets continued to flood in. At 11:30 AM, Loom took the drastic measure of blocking all traffic to loom.com via WAF, returning a 503 Service Not Available error. Service was restored at 2:45 PM, after an incident duration of 4 hours and 21 minutes. The impact was significant: 3,000 users were impacted, and a complete database and cache rollback was necessary, resulting in the loss of 3,000 Loom videos.

Post-incident, several technical remediations were implemented:

• The SameSite cookie attribute was changed to Lax, a fix that took three days to implement.
• Defense-in-depth measures were enhanced, including low-scope cookies and device binding using historical IPs and user agents to flag unusual access patterns.
• Code audits were intensified, and security became a more active participant in quarterly planning.

Loom's bug bounty program, hosted on HackerOne, played a crucial role in continuous security testing. Over $170,000 was paid out across 250+ triaged reports, with the highest single bounty reaching nearly $8,000. A recurring critical issue identified through the program was email verification bypasses, attributed to rapid development velocity and legacy code. The program's success was maintained through annual bounty increases, short-term promotions (e.g., 1.5x or 2x bounties), and long-term incentives (e.g., $555 for five valuable vulnerabilities on one asset, or specific bounties for AI security issues). High efficiency in triage and payment (96% of reports met fast payment criteria) was key to researcher engagement.

For SAST (Static Application Security Testing), Loom adopted a unique approach. Instead of relying solely on generic rules, they focused on custom rules derived from post-mortems of past incidents and recurring security issues. These rules were integrated as inline PR comments within the development workflow, providing developers with context-specific guidance without hindering velocity.

Infrastructure security was tightly integrated, with the security team reporting under the same manager as infrastructure. They utilized Infrastructure as Code (IaC) tools like Terraform with integrated security linter checks. A CSPM (Cloud Security Posture Management) product was employed to identify misconfigurations, and Service Control Policies (SCPs) were extensively used to enforce guardrails and prevent the creation of insecure resources in the first place.

Demo / Proof of Concept

▶ Watch: SameSite=None vs. Lax Impact on Background Requests (17:00)

The talk primarily focused on recounting Loom's security journey, including strategic decisions, process implementations, and a detailed post-mortem of a significant security incident. While the speakers described the functionality of various security tools and the impact of technical changes, no live demonstration or proof of concept was presented during the session.

Defensive Implications

▶ Watch: Service Control Policies (SCPs) for Insecure Resource Prevention (29:00)

The Loom security journey offers several critical defensive implications for organizations striving to build and maintain a robust security posture:

• Meticulous WAF and CDN Configuration: Organizations must exercise extreme caution when configuring Web Application Firewalls (WAFs) and Content Delivery Networks (CDNs). Understand how managed policies (e.g., AWS caching optimized) can override explicit settings like Cache-Control: max-age=0. For GraphQL APIs, WAF rules need to be granular, targeting specific operations and queries rather than broad endpoints. Thorough testing in environments that accurately simulate production traffic is essential to prevent self-DDoS or blocking legitimate users.
• Cookie SameSite Attribute Best Practices: The incident highlighted the critical importance of the SameSite cookie attribute. Prioritize SameSite=Lax for most cookies to prevent cross-site request forgery (CSRF) and session leakage, especially for background requests fetching static assets. While SameSite=None is necessary for cross-site embedding, its security implications must be fully understood and mitigated with other controls. Avoid compromising security for short-term functional gains without robust compensating controls.
• Defense-in-Depth for Session Management: Implement multiple layers of security for user sessions. This includes low-scope cookies (limiting their reach), device binding (using historical IP addresses and user agents to detect anomalous logins), and multi-factor authentication. These measures can significantly reduce the blast radius of session-related vulnerabilities.
• Proactive Vulnerability Management: Establish a clear vulnerability management policy with standardized scoring (e.g., CVSS), defined SLAs, and mechanisms for leadership accountability. Empower the security team to own and implement compensating controls to manage risks effectively when immediate fixes are not feasible.
• Strategic Security Tooling and Customization: Adopt a hybrid approach to security tooling, combining commercial products with in-house customizations. Focus on building custom rules for SAST and threat detection platforms based on internal incidents and recurring issues. This provides highly relevant and actionable feedback to developers, moving beyond generic OWASP Top 10 checks.
• Robust Incident Response and Post-Mortem Culture: Develop comprehensive incident response plans with clear severity definitions (e.g., SE0, SE1) and ownership. Crucially, foster a culture of transparency and "fail forward" in post-incident analysis. Detailed post-mortems, like the one shared by Loom, are invaluable for identifying root causes, implementing lasting fixes, and preventing recurrence. Be prepared for drastic measures, such as temporarily taking services offline, if necessary to contain a critical incident.
• Integrate Security into Development Workflows: Embed security into the entire software development lifecycle. This includes integrating custom SAST rules as inline PR comments, involving security in quarterly planning, and implementing "trust but verify" mechanisms like break-glass scenarios with audited access to sensitive systems.
• Leverage Bug Bounty Programs Effectively: A well-managed bug bounty program, like Loom's on HackerOne, can be a powerful tool for continuous security testing. Incentivize specific vulnerability types, maintain high efficiency in triage and payouts, and learn from the broader security community to keep the program effective and engaging for researchers.
• Strengthen Infrastructure Security with IaC and CSPM: For organizations using Infrastructure as Code (IaC) (e.g., Terraform), integrate security linters and automated checks into the deployment pipeline. Utilize Cloud Security Posture Management (CSPM) products to continuously monitor for misconfigurations. Implement Service Control Policies (SCPs) extensively to enforce guardrails at the organizational level, preventing the creation of insecure resources.
• Cultivate a Security-Conscious Culture: Ultimately, security is a shared responsibility. Foster a culture where developers are actively engaged, care about security, and collaborate closely with the security team. Transparency, optimism, and a willingness to learn from failures are foundational to building a resilient security program.

Key Takeaways

• Phased Security Program Development: Building a security program in a fast-growing startup requires a pragmatic, phased approach, starting with foundational elements like WAF deployment and gradually scaling processes, tools, and automation.
• WAF and CDN Configuration Complexity: WAF and CDN configurations are highly nuanced; misconfigurations, especially regarding cookie handling and caching policies (e.g., AWS managed policies overriding explicit settings), can lead to severe issues like self-DDoS, legitimate traffic blocking, or critical session leakage.
• The Criticality of SameSite Cookies: The SameSite cookie attribute is a fundamental security control. Understanding its interaction with different request types (e.g., background fetches vs. top-level navigation) and CDN caching is paramount to prevent session hijacking and cross-site vulnerabilities.
• Value of a Mature Bug Bounty Program: A well-managed bug bounty program, characterized by consistent incentives, efficient triage, and continuous learning from the community, is an invaluable asset for proactive vulnerability discovery and maintaining a strong security posture.
• Customized SAST for Contextual Guidance: Moving beyond generic SAST rules to develop custom rules based on internal incident post-mortems and recurring issues provides highly relevant and actionable security feedback directly within developer workflows (e.g., inline PR comments), fostering better security practices.
• Security as a Shared Engineering Responsibility: Effective security at scale is deeply intertwined with engineering culture, requiring strong cross-functional collaboration, transparent communication, integration into quarterly planning, and a "fail forward" mindset where developers actively participate in and care about security.

About the Speaker(s)

Narayan Gowraj leads security at Loom, an asynchronous messaging app that was recently acquired by Atlassian. Prior to his role at Loom, Narayan focused on product security and cloud security at Left. He joined Loom in 2021 as the founding security engineer, spearheading the development and scaling of the company's security program during a period of significant growth and ultimately through its acquisition.

Nishant Jain is a Security Engineer at Loom. He joined the company in 2023 as a new graduate, making Loom his first professional role. Nishant's primary responsibilities include application security (AppSec), managing the bug bounty program, and conducting security research. He brings prior experience as a bug bounty hunter, which informs his approach to triaging reports and building relationships with security researchers.