Project Lightning Talk: A Security Engineer’s Guide to TAG Security - Brandt Keller, Technical Lead

Project Lightning Talk: A Security Engineer’s Guide to TAG Security

Speakers: Brandt Keller; Technical Lead

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=PcRORHC1NYY

Overview

In an era where the cloud-native landscape is expanding at an unprecedented rate, securing its foundational projects has become a critical challenge that necessitates a collective and scalable approach. Brandt Keller, a Technical Lead for the Technical Advisory Group Security (TAG Security), delivered a compelling lightning talk at KubeCon EU, emphasizing the vital role of TAG Security within the Cloud Native Computing Foundation (CNCF) ecosystem. His presentation served as a clarion call for increased accessibility and diversity in security contributions, highlighting how a broad spectrum of roles can actively participate in fortifying cloud-native projects.

Keller underscored the reality that security strategies effective over the past decade may prove inadequate for the next, given the accelerated pace at which new projects join the CNCF landscape. Many projects, particularly smaller ones or those in early stages, often find themselves ill-equipped to handle complex security incidents like CVE submissions or to implement robust security architectures. TAG Security steps in to fill this gap, offering a suite of services from strategic advisory to hands-on security assessments, all aimed at ensuring minimum security thresholds are met across the CNCF portfolio and fostering end-user confidence.

Beyond its direct services, a core message of Keller's talk was the imperative to cultivate a sustainable security posture across the entire cloud-native ecosystem. This involves not only formal assessments and advisories but also fostering a culture of shared responsibility and continuous learning. By lowering the barriers to entry for security involvement, TAG Security aims to empower a more diverse group of contributors, transforming security from a specialized bottleneck into a widely distributed and collaborative effort that can truly scale with the growth of cloud-native technologies.

Background

▶ Watch: Introduction and increasing diversity in security roles (0:00)

The rapid evolution and expansion of the cloud-native ecosystem present both immense opportunities and significant security challenges. As Brandt Keller articulated, the sheer volume of projects entering the CNCF landscape means that traditional security approaches, often reliant on a small cadre of experts, are no longer sustainable. Projects frequently encounter security-related issues—ranging from vulnerability disclosures and CVE (Common Vulnerabilities and Exposures) submissions to fundamental questions about secure design—without adequate internal resources or clear guidance on how to respond. This creates a critical need for a centralized, yet adaptable, support mechanism.

This problem is exacerbated by the inherent diversity of cloud-native projects. Each project, with its unique architecture, technology stack, and development practices, presents a nuanced security posture. A one-size-fits-all approach to security assessments or guidance is often ineffective. Furthermore, project maintainers and contributors, while experts in their respective domains, may lack specialized security expertise, leaving them vulnerable to sophisticated threats or struggling to implement best practices. The absence of readily available, accessible security support can impede project maturity, delay adoption, and ultimately erode end-user confidence in the security of the broader cloud-native ecosystem.

To address these challenges, the CNCF established various Technical Advisory Groups (TAGs), with TAG Security specifically mandated to focus on the security aspects of cloud-native projects. TAG Security's mission is multifaceted, encompassing:

• Advising the TOC (Technical Oversight Committee): TAG Security frequently advises the TOC on security requirements for projects seeking to join the sandbox or graduate to incubating stages. This involves defining minimum security thresholds, implementing checks and balances, and ensuring the overall security integrity of the landscape. They provide guidance on architectural considerations and design principles to help projects build security in from the ground up, ultimately enhancing end-user confidence in project consumption.
• Conducting Assessments: Recognizing the diverse needs of projects, TAG Security offers various assessment methodologies. These range from self-assessments, which empower projects to conduct initial security evaluations, to more in-depth joint assessments where TAG Security members collaborate directly with project teams. These assessments aim to identify vulnerabilities, suggest improvements, and validate security postures.
• Facilitating Research and Documentation: Through its working groups, TAG Security engages in broader research initiatives, publishing documentation and white papers. These resources contribute to a collective body of knowledge, disseminating best practices, security patterns, and guidance across the entire ecosystem, thereby enhancing overall security maturity.

Despite these existing mechanisms, Keller emphasized that awareness and involvement remain key hurdles. Many projects and potential contributors are simply unaware of the support available or perceive high barriers to entry for security contributions. The talk's central theme, therefore, was to bridge this gap, promoting greater accessibility and diversity of involvement to ensure the long-term sustainability and security of the cloud-native landscape.

Key Findings

▶ Watch: TAG Security's advisory role for project security (1:20)

Brandt Keller's presentation, while not detailing specific technical vulnerabilities or exploits, unveiled critical insights into the state of security within the cloud-native ecosystem and the strategic direction required to address its challenges. The primary "findings" are not discoveries in the traditional research sense, but rather a set of observations, identified needs, and strategic imperatives for the future of cloud-native security.

1. The Unsustainable Growth vs. Security Gap: A fundamental finding is the widening gap between the rapid scaling of the cloud-native landscape and the capacity of existing security paradigms to secure it effectively. Keller explicitly stated, "What's worked for the last 10 years probably won't work for the next 10 years." This highlights an urgent need for new, scalable approaches to security that can keep pace with the influx of new projects and evolving threat landscapes. Traditional, centralized security teams are simply insufficient to manage the security posture of hundreds of diverse projects.

1. Security as a Shared Responsibility: The talk firmly established that security is not solely the domain of specialized security engineers or dedicated teams. Instead, it is a shared responsibility that spans every role within the ecosystem—from core contributors writing code, to project maintainers overseeing development, and even to end-users consuming the projects. This finding shifts the paradigm from a siloed security function to an integrated, community-wide effort.

1. The Critical Need for Accessibility and Diversity: Keller identified a significant opportunity to enhance security outcomes by fostering greater accessibility and diversity of involvement. Many individuals possess valuable skills that can contribute to security, but may perceive high barriers to entry. By actively seeking to involve a broader range of participants—regardless of their primary role or formal security background—the ecosystem can tap into a richer pool of perspectives and expertise, leading to more robust and innovative security solutions. This also includes lowering the perceived barrier for learning and contributing, making security tasks more approachable for newcomers.

1. Significant Opportunities for Skill Development: A powerful finding for individual contributors is the immense potential for skills development through engagement with TAG Security. Participating in activities like security assessments, threat modeling, and architectural reviews provides practical, real-world experience directly applicable to everyday enterprise security challenges. This creates a symbiotic relationship where contributors enhance their professional skills while simultaneously strengthening the security of the cloud-native landscape. Keller emphasized the "learn from others" aspect, where individuals can develop their security acumen by working alongside experienced practitioners.

1. Building End-User Confidence and Reducing Risk: Ultimately, the collective efforts facilitated by TAG Security contribute directly to two critical outcomes: building end-user confidence and significantly reducing risk across the landscape. When projects undergo rigorous security assessments and adhere to best practices, end-users can consume these technologies with greater assurance. This confidence is vital for broader adoption and the long-term sustainability of the entire CNCF ecosystem, preventing security concerns from becoming an impediment to innovation.

1. The "Chat Loop Back Off" Approach to Assessments: While not a "finding" in the traditional sense, Keller highlighted a crucial methodological insight for effective security assessments: the "chat loop back off approach." This refers to approaching each project with a fresh, unbiased perspective, as if knowing "nothing about this project." This method ensures that assessments are thorough, challenge assumptions, and uncover nuanced security considerations unique to each project, rather than relying on generalized patterns that might overlook critical details.

These findings collectively paint a picture of an ecosystem at a critical juncture, where scaling security effectively requires not just technical solutions, but a profound shift towards collaborative, accessible, and diverse community engagement.

Technical Deep Dive

▶ Watch: Embracing accessibility and diversity in security involvement (3:45)

While Brandt Keller's talk focused on the strategic and community aspects of cloud-native security, it implicitly detailed several critical technical processes and architectural considerations championed by TAG Security. The "technical deep dive" here pertains less to specific exploits or code snippets, and more to the methodologies, frameworks, and principles that TAG Security employs to enhance the security posture of projects within the CNCF.

At its core, TAG Security acts as an expert resource for projects navigating the complex security landscape. Their involvement begins early, often during a project's entry into the CNCF sandbox, where they advise the Technical Oversight Committee (TOC) on essential security requirements. This advisory role is crucial for establishing minimum security thresholds—a set of baseline security standards that projects must meet to ensure a foundational level of resilience against common threats. These thresholds encompass areas such as secure coding practices, dependency management, vulnerability disclosure policies, and basic hardening configurations. By integrating security checks and balances into the project lifecycle, TAG Security helps prevent insecure projects from progressing without proper remediation.

A significant part of TAG Security's technical contribution lies in its assessment methodologies. These are structured approaches designed to systematically identify and mitigate security risks:

1. Self-Assessment: This is a crucial first step for many projects. The self-assessment framework provides projects with a self-service project assessment for security. While the talk did not detail the exact questions or tools, it implies a structured questionnaire or checklist that guides project teams through an evaluation of their own security practices. This typically covers areas such as:

• Vulnerability Management: How are vulnerabilities identified, reported, and patched?
• Access Control: Who has access to repositories, build systems, and deployment environments? How is this managed?
• Supply Chain Security: How are third-party dependencies managed and secured? What build integrity checks are in place?
• Secure Development Lifecycle (SDL) Integration: Are security considerations integrated into the design, development, testing, and deployment phases?
• Incident Response Plan: Is there a defined process for responding to security incidents?

This self-assessment process is not merely a compliance exercise; it's designed to educate project teams and enhance their understanding of security best practices, ultimately boosting their confidence and that of the CNCF and TOC in the project's security posture.

1. Joint Assessment: For more in-depth evaluations, TAG Security conducts joint assessments, which are highly collaborative and hands-on. This involves TAG Security maintainers and contributors working "hand in hand" with project teams. The technical depth here is significant, often involving:

• Threat Modeling: A systematic process to identify potential threats and vulnerabilities in a system's design. This involves breaking down the application or system, identifying trust boundaries, data flows, and potential attack surfaces. Methodologies like STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) are commonly employed in such exercises.
• Architecture Review: A critical examination of the project's architectural design from a security perspective. This includes evaluating the security of inter-component communication, data storage, authentication mechanisms, authorization schemes, and overall system resilience. TAG Security advises on secure design patterns and helps projects avoid common architectural pitfalls.
• Code Review (Conceptual): While not explicitly stated as line-by-line code review, the "in-depth personnel" involvement and focus on "responding to the security review of different changes to the systems" implies a level of scrutiny that can include reviewing security-critical components or design patterns in code.
• Configuration Review: Assessing the security of project configurations, including CI/CD pipelines, container images, Kubernetes manifests, and cloud infrastructure settings.

A key aspect of these joint assessments, as highlighted by Keller, is the "chat loop back off approach"—approaching each project with an open mind, acknowledging its unique nuances, and learning from its specific implementation rather than imposing generic solutions. This ensures that the security advice is tailored and maximally effective.

Furthermore, TAG Security's research working groups contribute to the technical landscape by publishing documentation and white papers. These resources often delve into specific security challenges relevant to cloud-native technologies, such as secure container image practices, Kubernetes hardening guides, or best practices for managing secrets. By disseminating this knowledge, TAG Security ensures that the collective security expertise benefits the entire ecosystem, promoting the adoption of technically sound security solutions. The goal is to make these patterns and solutions widely accessible, reducing the barrier for other projects to implement robust security.

In essence, TAG Security's technical contributions are less about developing new security tools and more about establishing, promoting, and assisting with the application of robust security engineering principles and methodologies across the vast and varied cloud-native landscape. They guide projects through the practical application of security, from initial design considerations to ongoing vulnerability management, ensuring that "sustainability of security" is not just a goal, but an achievable reality.

Demo / Proof of Concept

▶ Watch: Skill development opportunities through security contributions (4:30)

This talk focused on strategic initiatives, community engagement, and the operational frameworks of TAG Security rather than a technical demonstration or proof of concept. Brandt Keller's presentation outlined the services, methodologies, and collaborative opportunities within TAG Security, emphasizing process and participation over live code or exploit showcases. There were no specific tools, exploits, or architectural implementations demonstrated during the session.

Defensive Implications

▶ Watch: How to get involved with TAG Security (6:15)

The insights shared by Brandt Keller regarding TAG Security's mission and methodologies carry significant defensive implications for all stakeholders within the cloud-native ecosystem. Successfully navigating the rapidly expanding threat landscape requires a proactive, collaborative, and integrated security strategy, and TAG Security provides a blueprint for how this can be achieved.

For Cloud-Native Project Maintainers and Contributors:

1. Proactive Engagement with TAG Security: The most direct defensive implication is to actively leverage TAG Security's resources. Projects, especially those in the sandbox or incubating stages, should proactively seek out TAG Security for advisory services on architecture and design. Engaging in self-assessments is a crucial first step to identify basic security gaps, while requesting joint assessments provides an invaluable opportunity for in-depth, expert-led security reviews, including threat modeling and architectural evaluations. This proactive stance helps embed security early in the development lifecycle, which is far more cost-effective than remediating vulnerabilities post-deployment.
2. Integrate Security into the SDLC: Maintainers should adopt a mindset where security is a continuous consideration, not an afterthought. This means incorporating security requirements into design specifications, conducting regular code reviews with a security lens, implementing automated security testing in CI/CD pipelines, and establishing clear vulnerability disclosure and response processes. TAG Security's guidance on meeting "minimum thresholds" should be seen as a baseline for a secure Software Development Lifecycle (SDLC).
3. Embrace Shared Responsibility: Recognize that security is a collective effort. Encourage all contributors to consider security implications in their work, from writing secure code to contributing to security documentation. Foster a culture where security issues are reported, discussed, and addressed openly.
4. Continuous Learning and Skill Development: The opportunity for skill development through TAG Security involvement is a defensive asset. Maintainers and contributors can learn practical security skills like threat modeling, secure design patterns, and vulnerability management, which directly enhance the security posture of their projects.

For Cloud-Native End-Users and Consumers:

1. Prioritize Secure Projects: End-users should consider the security posture of cloud-native projects as a critical factor in adoption. Look for projects that have demonstrably engaged with TAG Security, undergone assessments, or publicly share their security practices and vulnerability disclosure policies. This due diligence helps reduce the risk of integrating insecure components into their own systems.
2. Demand Transparency and Accountability: Encourage projects to be transparent about their security efforts. A project's willingness to engage with TAG Security, publish assessment results (where appropriate), and have clear processes for handling security issues is a strong indicator of its commitment to security.
3. Understand Shared Risk: While projects bear the primary responsibility for their own security, end-users also have a role in securing their deployments and reporting potential issues. Understanding the security characteristics of the components they use is crucial for effective risk management.

For the Broader CNCF Ecosystem and Security Professionals:

1. Invest in Scaling Security Initiatives: The CNCF, through TAG Security, must continue to invest in and scale its security initiatives to match the accelerating growth of the landscape. This includes developing more accessible tools, expanding educational resources, and fostering a larger pool of security contributors.
2. Promote Diversity and Accessibility: Actively work to lower the barriers to entry for security contributions. This involves creating welcoming environments, providing mentorship, and making security knowledge and tasks more approachable for individuals from diverse backgrounds and skill sets. The "accessibility and diversity of involvement" is not just a social good; it's a strategic imperative for comprehensive security.
3. Standardize and Share Best Practices: TAG Security's role in research and documentation is vital for establishing and disseminating best practices. The ecosystem benefits when security patterns, reference architectures, and threat models are shared widely, allowing projects to learn from each other's experiences and avoid common pitfalls. This ensures a consistent baseline of security across the landscape.
4. Foster a Culture of Continuous Improvement: Security is an ongoing journey. The defensive posture of the cloud-native ecosystem must evolve continuously, driven by lessons learned from assessments, new research, and emerging threats. TAG Security facilitates this by promoting a culture of learning and adaptation.

In summary, the defensive implications of Brandt Keller's talk revolve around a paradigm shift: from reactive, siloed security to a proactive, collaborative, and continuously improving security posture driven by a diverse and engaged community, all facilitated and guided by the expert resources of TAG Security.

Key Takeaways

• Shared Security Responsibility: Security in the cloud-native landscape is a critical, shared responsibility across all roles—contributors, maintainers, and end-users—necessitating collective engagement to scale effectively.
• TAG Security's Core Services: The Technical Advisory Group Security (TAG Security) provides essential services including advising the TOC, conducting self-assessments and joint assessments, and publishing research to secure CNCF projects.
• Need for Accessibility and Diversity: There is an urgent and significant opportunity to lower the barriers to entry for security contributions, actively seeking a greater diversity of involvement to strengthen the ecosystem.
• Practical Skill Development: Engaging with TAG Security initiatives, such as threat modeling and architectural reviews, offers invaluable real-world experience and fosters practical security skill development for participants.
• Building Confidence and Reducing Risk: Proactive engagement with TAG Security's resources helps projects meet minimum security thresholds, significantly reduces overall risk, and builds crucial end-user confidence in cloud-native technologies.
• Sustainability Through Collaboration: Ensuring the long-term security and sustainability of the cloud-native ecosystem depends on a collaborative, continuously learning community that scales its security efforts to match rapid project growth.

About the Speaker(s)

Brandt Keller is a Technical Lead for TAG Security, the Technical Advisory Group focused on security within the Cloud Native Computing Foundation (CNCF). He is deeply passionate about security, describing it as something that "really gets you pumped up." Keller is a strong advocate for increasing accessibility and diversity of involvement in security contributions across the cloud-native landscape, believing that a broader range of perspectives is crucial for scaling security efforts. His work involves advising the TOC, conducting security assessments, and fostering collaborative learning, approaching each project with an open mind to understand its unique security nuances.