Evolving Secure Development through FedRAMP Continuous Monitoring Trends

Evolving Secure Development through FedRAMP Continuous Monitoring Trends

Speakers: Stephanie Harris, Principal Product Security Engineer, Red Hat; Christopher Lusk, Principal Product Security Engineer, Red Hat

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=BRcZfE00VN8

Overview

This talk by Stephanie Harris and Christopher Lusk from Red Hat delves into the intricate world of FedRAMP (Federal Risk and Authorization Management Program) continuous monitoring and its profound impact on evolving secure development practices within a large enterprise. The speakers, both principal product security engineers deeply embedded in Red Hat's FedRAMP program, share their journey of navigating stringent government compliance requirements, particularly for a FedRAMP High categorized system. Their presentation highlights how the continuous, data-intensive nature of FedRAMP compliance can be leveraged not just to meet regulatory obligations, but to drive significant, measurable improvements in an organization's overall security posture and development lifecycle.

The core of their discussion revolves around the challenges encountered during two years of continuous monitoring, including scaling issues, a surge in vulnerabilities, and the limitations of manual processes. By meticulously analyzing vulnerability data collected through FedRAMP scanning, Harris and Lusk identified critical trends in affected components and common weakness enumerations (CWEs). This data-driven approach allowed them to propose concrete, actionable suggestions for enhancing secure development, ranging from implementing automated image blocking in pipelines to refining internal policies and training. The talk ultimately underscores how compliance efforts, often perceived as burdensome, can become a powerful catalyst for organizational-wide security improvements, fostering a "rising tide lifts all boats" effect across Red Hat's product offerings.

Background

▶ Watch: Introduction and talk agenda overview (0:00)

The foundation of this talk lies in understanding FedRAMP and its associated continuous monitoring requirements. FedRAMP is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services used by federal agencies. Its inception was driven by the Federal Information Security Management Act (FISMA), which tasked NIST with creating special publications to guide agencies in meeting security requirements. The primary guide for FedRAMP is NIST Special Publication (SP) 800-53, a comprehensive catalog of security controls for information systems.

For a Cloud Service Provider (CSP) like Red Hat, the FedRAMP journey is rigorous. It begins with an agency sponsor and a kickoff meeting with the FedRAMP Program Management Office (PMO). This is followed by a multi-month security assessment, typically three to four months long, conducted by a Third-Party Assessment Organization (3PAO). This assessment involves extensive testing, including scans, penetration testing, and manual interviews. Upon successful completion, the agency grants an Authorization to Operate (ATO), which then undergoes review by the FedRAMP PMO before the offering is listed on the FedRAMP Marketplace. Red Hat's own journey for their FedRAMP High system, starting in April 2022, culminated in an ATO in December 2022, with subsequent PMO approval taking an additional six months.

However, obtaining an ATO is not the end; it marks the beginning of continuous monitoring. This ongoing process requires CSPs to perform regular tasks, some constantly, others annually, depending on the system's categorization level (Low, Moderate, or High). Red Hat's system is classified as FedRAMP High, mandating the most stringent and frequent monitoring activities, ranging from constant to annual. These tasks include pulling and reviewing inventory, managing user access with least privilege principles, reviewing change management for significant alterations, and monitoring ports, protocols, and services for least functionality. According to the speakers, the most resource- and time-consuming aspect of continuous monitoring is vulnerability scanning and subsequent management.

Red Hat was approximately two years into continuous monitoring when they initiated this study, prompted by several critical challenges. Their scope had tripled in size, making existing manual processes unscalable. They observed a significant increase in CVEs, often hundreds at a time, with limited engineering resources for remediation. Furthermore, questionable image hygiene was noted, with teams deploying images containing hundreds of CVEs into the FedRAMP boundary. These issues spurred the team to analyze their data for trends, aiming to prevent vulnerabilities from entering the environment or appearing on scans in the first place.

The challenges were categorized into three areas:

1. Overall Continuous Monitoring: Building a program from the ground up, heavy reliance on manual processes and spreadsheets, internal evangelization of FedRAMP to engineering teams unfamiliar with its requirements, and a stringent process for evaluating and approving new security tooling within the FedRAMP environment.
2. Image Health: While the average number of vulnerabilities typically hovered around 20-30 for a smaller platform, it rose to 70-80 with the tripled scope, with alarming spikes of 300-400 CVEs in some images. Research with engineers revealed multifaceted issues: delays in image health grades from the CLA scanner (which assesses images stored in Red Hat's Quay repository before they enter the boundary), and operational concerns about blocking images based on these grades, as reclassified CVEs could unpredictably change an image's health status.
3. Analyzing Vulnerability Trends: The FedRAMP High environment is isolated from Red Hat's commercial environment, meaning data from scans required sanitization and manual processing (extensive spreadsheet work) before it could be analyzed for trends and metrics. This isolation prevented the use of integrated scanning tools that could provide automated health metrics.

Key Findings

▶ Watch: The abbreviated FedRAMP authorization process (2:55)

The comprehensive analysis undertaken by Harris and Lusk spanned from January 2023 to January 2024, encompassing their first year of continuous monitoring. During this period, they observed a total of 1,500 CVEs across 220 components. Notably, this count excluded 673 kernel vulnerabilities that were dropped during the year to prevent data skew, allowing for a focused analysis on non-kernel issues.

From this extensive dataset, three key "top 10" lists were generated to highlight recurring patterns:

1. Top 10 Most Commonly Affected Components: These components collectively accounted for approximately 42% of all CVEs identified within the FedRAMP environment during the two-year period. When categorized, the breakdown of vulnerability sources was revealing:

• Programming Languages: Over 53% of vulnerabilities.
• CLI Tools: 30% of vulnerabilities.
• Libraries: 10% of vulnerabilities.
• Cryptographic Modules (specifically OpenSSL): Approximately 5.9% of vulnerabilities.

1. Top 10 Most Common CWEs: This list aimed to connect the observed vulnerabilities to underlying weaknesses, providing actionable insights for secure development. While the full list was not read aloud, the speakers emphasized that many were related to memory-based weaknesses. They acknowledged that some of the identified CWEs were "class-level" categories, which MITRE discourages mapping directly due to potential for misleading information. However, they included them with the intention of conducting deeper dives later to pinpoint more specific actionable data. The presence of these CWEs pointed towards areas where secure development practices and rigorous testing could yield significant improvements.

1. Top 10 Most Common CWEs on POAM: The POAM (Plan of Action and Milestones) is a FedRAMP acronym for vulnerabilities that have passed their remediation deadlines. Unsurprisingly, the CWEs appearing most frequently on the POAM were largely redundant with those in the general "Top 10 Most Common CWEs" list, differing only in their specific order. This indicated that the same underlying weaknesses were not only prevalent but were also proving challenging to remediate within the prescribed timelines.

These findings provided a crucial data-backed foundation for Red Hat's internal discussions on evolving secure development, moving beyond assumptions to focus on empirically identified problem areas.

Technical Deep Dive

▶ Watch: Continuous monitoring begins: you're never done! (4:15)

The technical underpinnings of Red Hat's FedRAMP environment and its continuous monitoring program are complex, navigating stringent government regulations, specific tooling, and internal development practices.

At the core of FedRAMP compliance are the NIST SP 800-53 controls. Red Hat's FedRAMP environment, being FedRAMP High, is subject to over 400 controls. These controls, initially designed for on-premise environments, often require significant interpretation and education when applied to modern cloud-based services. This means auditors and government representatives frequently need detailed explanations to understand how Red Hat's cloud-native architecture and open-source components meet traditional control objectives.

Vulnerability scanning and configuration scanning are paramount in continuous monitoring. While vulnerability scanning identifies CVEs, configuration scanning ensures adherence to security baselines using standards like CIS Benchmarks, STIGs (Security Technical Implementation Guides), and DSA (Defense Security Assessments). This constant cycle of scanning and attestation is a major resource consumer.

Red Hat's internal infrastructure plays a critical role. Container images are stored in Quay, their proprietary image repository. Before these images are pulled into the FedRAMP boundary, they undergo scanning by the CLA scanner, which assigns a "health index" or grade. A significant technical challenge identified was the delay in receiving these grades and the operational complexities of automatically blocking images based on them, especially given that CVE reclassifications could alter an image's health post-deployment.

A unique and particularly challenging aspect for Red Hat is maintaining FIPS (Federal Information Processing Standards) approval while simultaneously remediating vulnerabilities. FIPS 140-2 (and now 140-3) specifies cryptographic module security requirements. This creates a "tightrope" situation where moving to the latest commercial software versions might break FIPS compliance, even if those versions contain critical security fixes. The speakers detailed the NIST CMVP (Cryptographic Module Validation Program) process, which involves third-party testing and validation. Historically, this process was extremely rigid, preventing updates if the underlying operating system wasn't validated. However, recent changes have introduced an "update stream" for previously validated modules, offering more leniency for vulnerability remediation if approved by the agency. This allows for updates without undergoing a full re-validation, addressing a significant bottleneck.

The analysis of vulnerability trends itself presented a technical challenge. Due to the isolated nature of the FedRAMP High environment, direct integration with commercial-side tooling for automated data analysis was not possible. This necessitated a largely manual process, involving the sanitization of scan data and extensive use of spreadsheets to aggregate and analyze the 1,500 CVEs across 220 components.

Looking forward, Red Hat is actively working on technical solutions like Conflux, a secure pipeline discussed in a related presentation. Conflux aims to enable the creation of "secure profiles" that can, in fact, block images that do not meet specified health grades, addressing one of the core challenges identified in their image hygiene. This indicates a strategic shift towards embedding security controls directly into the build and deployment pipeline.

Demo / Proof of Concept

▶ Watch: Red Hat's challenges: scope, CVEs, manual processes, image hygiene (6:30)

The talk focused on presenting the findings of an extensive data analysis and outlining strategic suggestions for improving secure development practices. Therefore, no live demonstration or proof of concept was conducted during this presentation. The speakers' emphasis was on the analytical process, the derived metrics, and the proposed strategic changes rather than showcasing a specific tool or exploit.

Defensive Implications

▶ Watch: Goal: Prevent issues by analyzing data and trends (7:50)

The detailed analysis and the challenges encountered during FedRAMP continuous monitoring have significant defensive implications for Red Hat and other organizations operating in highly regulated environments. The "rising tide lifts all boats" ethos encapsulates the broader positive impact of this stringent compliance work.

1. Improvements to Public-Facing CVE Pages: A direct byproduct of FedRAMP's rigorous documentation requirements has been a substantial improvement in Red Hat's public-facing CVE pages. The need to justify severity ratings that differ from NVD (National Vulnerability Database) for agency approval led to the publication of detailed explanations directly on CVE pages. Furthermore, statements outlining how SSP (System Security Plan) controls within the FedRAMP environment mitigate specific CVE risks are now also published. This enhanced transparency and contextual information benefits not only FedRAMP agencies but also commercial customers, allowing them to better assess actual risk in their controlled environments.

1. Refinement of Vulnerability Remediation Timelines: The urgent demands of FedRAMP compliance, particularly for quicker remediation of moderate and low-severity CVEs, have acted as a forcing function. This pressure has led to a refinement of existing vulnerability remediation timelines for Red Hat's upstream commercial offerings, making them more responsive and efficient.

1. Extended Product Support Life Cycles: The necessity of maintaining FIPS approval means that the FedRAMP environment cannot always simply upgrade to the latest commercial versions of software or operating systems. This constraint has indirectly led to an extension of product support life cycles for specific FIPS-validated components, ensuring continued security patching while maintaining compliance.

1. Organizational Restructuring and Resource Allocation: The scale and complexity of FedRAMP compliance have necessitated significant internal changes within Red Hat. This includes the creation of new teams, internal realignments of existing teams, and the allocation of additional tools and resources specifically to support the FedRAMP program. This demonstrates how a compliance mandate can drive substantial organizational evolution.

1. Strategic Secure Development Improvements: Based on the identified trends, the speakers proposed several actionable defensive strategies:

• Blocking Poor Images: Implementing secure pipelines, such as Conflux, to automatically block container images that do not meet a predefined health grade (e.g., B and above) before they enter production environments. This proactive measure aims to "shift left" security, preventing vulnerable code from being deployed.
• Enforcing Secure Development Policies: Strengthening the enforcement of Red Hat's existing secure development life cycle (SDLC) processes. This includes re-evaluating the leniency of exceptions and ensuring that critical or exploited vulnerabilities are remediated before code is pushed to production, even if the FedRAMP environment is downstream from commercial products.
• Better Testing and Code Reviews: Leveraging the identified CWE trends to implement more targeted testing and code reviews. This involves prioritizing resources to address vulnerabilities that pose the highest threat to the specific environment and integrating automation to reduce manual toil.
• Targeted Training: Developing and delivering more relevant secure development training. Instead of generic modules, the recommendation is for specific training on topics like image health, the repercussions of promoting vulnerable images, and how to address the memory-based weaknesses identified in the trend analysis.
• Accountability: Implementing mechanisms for accountability for teams that consistently skirt established security policies. This is seen as essential for driving lasting behavioral and process changes.

These defensive implications highlight a virtuous cycle where compliance requirements, despite their initial burden, ultimately lead to a more mature, robust, and proactive security posture across the entire organization.

Key Takeaways

• FedRAMP continuous monitoring is a perpetual and resource-intensive undertaking, particularly for FedRAMP High systems, with vulnerability scanning being a major time and resource consumer.
• Leveraging compliance-mandated data, even through manual means, can reveal critical and actionable trends in an organization's vulnerability landscape, identifying common affected components and underlying weaknesses (CWEs).
• Memory-based weaknesses are a significant and recurring source of vulnerabilities, underscoring the vital need for robust secure development practices, diligent code reviews, and comprehensive testing to address these fundamental issues.
• Stringent compliance frameworks like FedRAMP, while challenging, can act as a powerful catalyst for organizational-wide security improvements, leading to enhanced transparency in CVE reporting, refined remediation timelines, extended product support, and structural changes within engineering teams.
• Proactive measures such as implementing secure pipelines to block unhealthy images, enforcing secure development policies more rigorously, and providing targeted security training are crucial for evolving secure development practices and mitigating recurring vulnerability patterns.
• Navigating the complexities of compliance, such as balancing FIPS approval with vulnerability remediation, requires flexibility, education, and strong communication with auditing bodies and agencies to ensure security objectives are met without compromising operational stability.

About the Speaker(s)

Stephanie Harris is a Principal Product Security Engineer for the product security compliance team at Red Hat. With over three years in her current role, she has served as the Information System Security Manager (ISSM) for Red Hat's FedRAMP program, deeply involved in navigating the program's stringent requirements. Her background includes extensive experience in government, which provides her with a unique perspective on the intersection of federal regulations and open-source technology.

Christopher Lusk is also a Principal Product Security Engineer at Red Hat, having been with the company for about two and a half years. Throughout his tenure, he has primarily focused on FedRAMP initiatives. In addition to his FedRAMP responsibilities, Christopher has contributed to vulnerability management efforts supporting other compliance standards such as PCI DSS, SOC 2, and ISO, showcasing a broad expertise in security compliance and risk management.