LLMs at the Core: From Attention to Action in...

LLMs at the Core: From Attention to Action in...

Speakers: Fotis Chantzis, Paul McMillan

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=WQIFDnu-c6g

Overview

This technical article delves into the practical applications of Large Language Models (LLMs) in enhancing cybersecurity workflows, as presented by Fotis Chantzis and Paul McMillan from OpenAI at BSidesSF 2024. The talk, titled "LLMs at the Core: From Attention to Action in...", addresses the pervasive challenge of "stretched" security teams grappling with limited human bandwidth and an ever-increasing volume of tasks. It posits that while traditional security tools remain indispensable, LLMs can significantly augment their effectiveness, leading to profound impacts on team efficiency.

The core message is that AI, specifically off-the-shelf LLMs, can act as a "caffeine pill" for security teams, enabling them to focus on critical threats by automating and streamlining various security "drudgery." The speakers emphasize a vendor-neutral approach, asserting that the techniques discussed are applicable across different LLM providers. They introduce and open-source three new tools developed at OpenAI—the SDLC Bot, the Triage Bot, and components for Bug Bounty Triage—demonstrating how LLMs can be leveraged to reduce security friction for developers, improve attacker detection, and accelerate incident response.

This presentation is particularly relevant for security professionals seeking innovative ways to scale their operations and improve their defensive posture in an era of rapid technological change. By showcasing real-world implementations and lessons learned, Chantzis and McMillan provide a compelling vision for how LLMs can transform security from a reactive, human-intensive process into a more proactive, AI-augmented defense mechanism, always with a crucial "human in the loop."

Background

▶ Watch: LLM statelessness and context window management explained (03:00)

The landscape of modern cybersecurity is characterized by an acute imbalance: security teams are perpetually "stretched," likened to a "yoga pose called The Eternal Scream," while the volume and complexity of threats continue to escalate. This scarcity of human bandwidth means that critical tasks, from evaluating new projects in a fast-paced Software Development Life Cycle (SDLC) to triaging an overwhelming number of security reports and sifting through voluminous logs, often overwhelm existing resources. The traditional approach, relying solely on human expertise and conventional tooling, struggles to keep pace.

OpenAI's own experience with the launch of ChatGPT underscored this challenge. Upon its release, the product attracted "millions of users" and, predictably, "many very excited attackers" (or "enthusiasts") who sought to exploit unauthorized APIs. This led to an exhausting "cat and mouse game" of bot detection and mitigation. In a notable anecdote, the team developed CatGPT, a small, "very dumb model" that only talked about cats, to perplex and deter these unauthorized users, highlighting a creative, if unconventional, early application of AI in defense.

This experience, coupled with the broader industry need, spurred OpenAI's security team to explore how LLMs could move beyond novelties like CatGPT to address more serious, practical security challenges. The underlying problem is not a lack of security tools, but rather the human capacity to effectively utilize them and manage the sheer volume of data and requests. The speakers contend that LLMs are not intended to replace existing security infrastructure but to augment it, enhancing workflows and significantly boosting team efficiency. This talk, therefore, is grounded in the necessity of finding scalable solutions to human-centric security problems, leveraging the unique capabilities of LLMs to process, analyze, and interact with information at a scale impossible for human teams alone.

Key Findings

▶ Watch: Prompt engineering lessons: praising models, numerical scores (09:00)

The presentation by Fotis Chantzis and Paul McMillan revealed several key findings regarding the practical application of LLMs in cybersecurity:

• LLMs Augment, Not Replace, Traditional Security Tools: A central tenet is that LLMs are not a panacea to replace existing security infrastructure but rather powerful augmentations that enhance workflows and improve team efficiency. They help security teams "focus on the real threats" by automating mundane tasks.
• Stateless Nature of LLMs and Context Management: Modern base LLM APIs are inherently stateless. They do not "remember" prior conversations. Instead, context is maintained by including as much of the previous dialogue as possible within each new request's prompt. When space runs out, older parts of the history are deleted to fit within the token limit.
• Impact on Security Workflows: LLMs can have a profound impact on a team's efficiency by:
• Helping developers go faster by reducing security friction.
• Enabling humans to focus on the most important parts of defense.
• Improving the way attackers are caught.
• Accelerating other forms of security "drudgery."
• Prompt Engineering Best Practices:
• Praise the Model: Telling the model it is an "expert cyber security engineer" (versus a mediocre one) yields "far better results." This form of positive reinforcement in prompts significantly improves response quality.
• Numerical Outputs for Quality: Models provide "much higher quality" responses when asked to output scores and numbers (e.g., a risk rating on a 0-10 scale) instead of descriptive sentences.
• Data Quality is Paramount: The more high-quality data provided to the model, the better its performance. It's crucial not to "just shove everything in there" but to ensure the context is high-quality and trustworthy.
• LLMs Excel at "Needle in a Haystack" Problems: LLMs are "consistently decent" at identifying critical information within large, noisy, and often "boring" datasets, such as log files, where human analysts might get tired or miss details.
• The Indispensable Human in the Loop: Despite their power, LLMs are "not infallible." They can "hallucinate," "miss things," and make mistakes. Therefore, always maintaining a human in the loop for critical decisions, reviews, and dispute resolution is non-negotiable.
• Open-Sourcing Practical Tools: OpenAI has open-sourced three new tools—the SDLC Bot, the Triage Bot, and components for Bug Bounty Triage—to enable other organizations to adopt these LLM-driven security enhancements.

These findings collectively underscore the transformative potential of LLMs in cybersecurity, provided they are implemented with an understanding of their capabilities, limitations, and best practices for interaction and oversight.

Technical Deep Dive

▶ Watch: Access Manager for dynamic authorization based on access patterns (13:00)

The technical core of the presentation revolved around demonstrating how LLMs, specifically off-the-shelf models, can be integrated into various security workflows to enhance efficiency and effectiveness. The speakers detailed the underlying mechanics of LLM APIs and then showcased several practical tools developed at OpenAI.

At a fundamental level, modern LLM APIs operate on tokens, which represent approximately three-quarters of a word. The model's knowledge and relationships between these tokens are encoded in model weights, typically stored on a GPU. When a user queries a model, it's a single request-response cycle: the query is tokenized, passed through the weights, and a response is generated. Crucially, the model is stateless; it doesn't inherently "remember" past interactions. To simulate memory, as seen in conversational agents like ChatGPT, the system includes as much of the prior conversation as possible in each new request's prompt. When the context window fills, older parts of the conversation are pruned to make space.

SDLC Bot

The SDLC Bot is designed to assist security teams in prioritizing which projects or features require human security review, addressing the challenge of limited human bandwidth in a fast-paced SDLC.

• Mechanism: The bot can optionally ask users basic security questions about a project. More powerfully, it scours and monitors important Slack threads and analyzes relevant design documents.
• Output: Based on the gathered context, the LLM assigns a risk rating (on a 0-10 scale) and a confidence level for its decision.
• Scenario Example: In the "WoofSpeak" company scenario, an initial design document for an inventory tracking system, accessible only via VPN, received a medium risk score of 4. When the design was updated to remove private links and expose components to the internet due to tight deadlines, the SDLC Bot, monitoring the design document, automatically updated the risk rating to a medium-high 7, flagging it for increased scrutiny.
• Prompt Engineering Insights: The team learned that models can "overfit on the initial questions" if provided as examples. Praising the model, e.g., telling it it's an "expert cyber security engineer," significantly improves response quality. Furthermore, asking for numerical scores (like 0-10) yields "far better results" than free-form text.
• Enhanced Version (Internal): An internal version of the SDLC Bot automatically monitors Slack channels, infers topics of interest from chat history, and flags security-relevant discussions, such as a request for an admin account without MFA.

Triage Bot

The Triage Bot aims to automate responses to common security-related requests, thereby saving valuable time for on-call engineers.

• Functionality: It can automatically redirect users to the correct team (e.g., IT for password resets) or provide direct answers.
• Contextual Awareness: An advanced feature allows the bot to leverage the Slack history of a channel to provide more detailed and context-aware responses. For instance, if a user reports a credential error accessing an Azure storage container, the bot might not only suggest using an internal access management tool but also, based on past resolutions, recommend restarting the local DNS service.

Access Manager

To streamline access provisioning, the team developed Access Manager, a tool that helps users find the correct permissions without needing to know specific group names or details.

• Mechanism: It uses available group descriptions and metadata to find conceptual matches for a user's query and explains them.
• Workflow: An engineer attempting to list storage contents receives an access error. Using the Access Manager CLI, they query the system, which suggests the appropriate group name and offers to send an access request on their behalf. The engineer's manager then receives a pending request, which they can approve or deny. Approved access is logged for an audit trail and can be set to expire.
• Dynamic Authorization (Experimentation): The team also experimented with dynamic authorization, where access decisions are based on multiple factors: user context (location, time, device, job role), resource context (sensitivity), and environmental context (network conditions). By analyzing the cosine similarity of access patterns between different roles (e.g., infrastructure engineer vs. security engineer), the system can identify anomalous requests that warrant more scrutiny, potentially automating grants for requests that align with normal patterns.

Bug Bounty Triage

Addressing the overwhelming volume of bug bounty reports (initially over 900 per day, with 3,500 invalid reports in the first week), LLMs were deployed to automate initial triage.

• Categorization: The model sorts reports into four categories using pre-written prompts:

1. Model Safety or Correctness: Reports about model behavior (e.g., harmfulness, factual errors) are routed to a separate ingestion mechanism.
2. Customer Support: Requests for product help, payment issues, or functional bugs without security impact are sent to the customer support team.
3. Out of Scope Bug Reports: Technical issues not relevant to the bug bounty program (e.g., missing server headers, SPF/DMARC records, server version strings). The prompt's out-of-scope list is kept in sync with the public program.
4. Security Report: Legitimate security issues (e.g., XSS, CSRF, subdomain takeovers) are sent directly to human triage.

• Failure Modes & Anti-Prompt Injection: For this use case, prompt injection is not a significant concern because the model is not making payment decisions or generating custom responses; it's merely categorizing and selecting from pre-written, vetted messages. The worst outcome is a miscategorized ticket.
• Evaluation: The team uses evals (known good questions/answers where the model grades its own responses) and human hand-scoring of initial reports (e.g., via CSV comparison) to assess accuracy.
• Future Enhancements (Experimentation): LLMs are being explored to improve report quality by prompting reporters for more information (e.g., full URLs, email addresses) and to summarize key points, extract reproduction steps, or suggest clarifying questions for human triagers.

Log Analysis and Incident Response

LLMs are also being applied to accelerate log analysis and incident response.

• Log Analysis: LLMs are "consistently decent" at finding "the needle in the haystack" within noisy and lengthy logs. An example showed the model successfully identifying a PowerShell one-liner that established a reverse shell within a user's bash history, prompting an alert to security.
• Incident Response Bot: This bot ingests alerts from a traditional SIEM. For specific alert types (e.g., accidental policy violations), it initiates a chat with the user to inform them of the potential problem and offer a chance to revert the change. For example, if an engineer accidentally shares a sensitive Google Drive document publicly, the bot engages them, asks their intention, and if the user tries to avoid the question, the bot "insists and doesn't mess around." The conversation is then summarized by the model and sent to the DNR (Detect and Respond) team for follow-up.

These detailed applications illustrate the breadth of LLM utility in modern security operations, from proactive SDLC integration to reactive incident response.

Demo / Proof of Concept

▶ Watch: LLMs for bug bounty triage, handling high volume reports (16:00)

The presentation included several practical demonstrations and proof-of-concept scenarios to illustrate the capabilities of the LLM-driven tools.

1. WoofSpeak Scenario for SDLC Bot: The core narrative involved a fictional company, "WoofSpeak," developing an inventory tracking system.

• Initially, the system's design document specified access only via the company's VPN, leading the SDLC Bot to assign a medium risk score of 4 with a certain confidence level.
• A subsequent update to the design, driven by tight deadlines, removed the requirement for private links, implying potential internet exposure. The SDLC Bot, continuously monitoring the design document, automatically detected this change and updated the risk score to a medium-high 7, triggering a higher priority for human review. This showcased the bot's ability to dynamically assess risk based on evolving project details.

1. SDLC Bot Interaction: A Slack-based interaction was shown where a user inputs basic project information (description, design document link, point of contact, go-live date). The bot then provides its risk rating and confidence score, demonstrating the user-friendly interface. An internal, enhanced version of the SDLC Bot was also mentioned, which automatically monitors Slack channels and flagged a conversation where an admin account without MFA was requested, highlighting its proactive detection capabilities.

1. Triage Bot in Action: The Triage Bot was demonstrated handling common user requests:

• A new engineer locked out of their computer asks for help; the bot automatically responds and redirects them to the IT team.
• Another engineer requests elevated permissions for a storage container; the bot redirects them to use "ghlas access," an internal access management tool.
• A more complex scenario involved a user reporting a credential error for an Azure storage container. The bot not only suggested using the access manager tool but also, drawing from past Slack history, offered an additional debugging step: restarting the local DNS service. This illustrated the bot's ability to provide context-rich, detailed assistance.

1. Access Manager CLI Workflow: The Access Manager was demonstrated via a command-line interface.

• An engineer attempts to list the contents of a storage account but receives an access denied error.
• They then use the access manager CLI to query what's wrong. The LLM-powered tool suggests the correct group name required for access and offers to send an access request on behalf of the user.
• The engineer's manager receives this request and approves it. After approval, the engineer can successfully rerun the CLI command and access the storage container, with the access grant logged for an audit trail and set to expire.

1. Bug Bounty Report Handling: A humorous example of a non-security-related bug bounty report (about "selling chicken") was shown. The bot politely informed the user that this was not the correct place to report such an issue, using a pre-written, carefully vetted response. This highlighted the bot's role in filtering irrelevant reports and guiding users appropriately without generating custom, potentially problematic, text.

1. Log Analysis for Attack Detection: A snippet of a user's bash history was presented, containing a PowerShell one-liner that established a reverse shell to a C2 server. The LLM was given a prompt to summarize the session and decide if it warranted a security alert. The model successfully identified the "malicious reverse shell commodity run in the middle of the session" and recommended alerting security, demonstrating its capability to find "the needle in the haystack" in noisy log data.

1. Incident Response Bot Interaction: The final demo showcased the Incident Response Bot.

• An engineer accidentally shares a sensitive Google Drive document publicly.
• The bot initiates a chat with the engineer, alerting them to the action and asking if it was intentional, giving them an opportunity to revert the change.
• When the user attempts to deflect the conversation (e.g., "what is your favorite color?"), the bot "insists and doesn't mess around," staying focused on the security issue.
• The entire conversation is then summarized by the model and sent to the DNR team for follow-up, providing valuable context for human responders.

These demonstrations collectively provided concrete evidence of how LLMs can be practically applied to automate, streamline, and enhance various critical security functions, making security teams more efficient and effective.

Defensive Implications

▶ Watch: LLMs for log analysis and incident response bot for user interaction (23:00)

The insights and tools presented by Fotis Chantzis and Paul McMillan offer significant defensive implications for cybersecurity teams looking to leverage LLMs. These implications span across the entire security lifecycle, from proactive development security to reactive incident response.

1. Enhanced SDLC Security Prioritization: The SDLC Bot directly addresses the challenge of limited security bandwidth by automating the initial risk assessment of new projects and features. Defenders can implement similar bots to monitor design documents and communication channels, dynamically assigning risk ratings and flagging high-risk items for human review. This ensures that scarce human expertise is focused on the most critical areas, preventing security issues from being baked into the architecture early on.

1. Automated Security Request Triage: The Triage Bot demonstrates how to offload routine security requests from on-call engineers. By automating responses, redirections, and even providing context-aware debugging suggestions (drawing from historical data), defenders can significantly reduce the operational burden, allowing engineers to concentrate on complex, novel threats. This improves response times for common issues and enhances user experience.

1. Streamlined and Auditable Access Management: The Access Manager offers a model for improving access control. Defenders can implement LLM-driven systems to help users discover appropriate permissions, automate access requests, and ensure an audit trail for all grants. The experimentation with dynamic authorization, using factors like user context and access patterns (cosine similarity), suggests a path toward more intelligent, risk-adaptive access control, where anomalous requests trigger greater scrutiny, while routine, low-risk requests can be automatically approved.

1. Efficient Bug Bounty Program Management: For organizations running bug bounty programs, LLMs provide a powerful mechanism to manage the overwhelming volume of reports. Defenders can deploy LLM-based triage systems to automatically categorize and filter out invalid, out-of-scope, or customer support-related reports. This allows human triagers to focus exclusively on legitimate security findings, drastically improving the efficiency and effectiveness of the program. Furthermore, LLMs can guide reporters to submit higher-quality information, reducing the back-and-forth and accelerating resolution.

1. Accelerated Threat Detection and Incident Response: LLMs excel at sifting through vast amounts of data, making them invaluable for log analysis. Defenders can use LLMs to summarize session transcripts and identify suspicious activities (e.g., reverse shells) that might be missed by human analysts due to fatigue or data volume. The Incident Response Bot provides a model for proactive engagement with users on potential policy violations (e.g., public sharing of sensitive documents), offering immediate intervention and gathering crucial context for the DNR team, thereby accelerating the initial phases of incident response.

1. Importance of Human Oversight and Data Quality: A critical defensive implication is the absolute necessity of maintaining a human in the loop. While LLMs augment, they are not infallible; they can "hallucinate" or miss critical details. Defenders must establish processes for human review, evaluation (evals), and dispute resolution to ensure accuracy and prevent false negatives or positives from impacting security posture. Furthermore, the effectiveness of these LLM-driven tools is directly tied to the quality of the data they are trained on and the context they are provided. Investing in high-quality, consistent data is a foundational defensive measure.

By strategically integrating LLMs into these areas, security teams can move towards a more proactive, efficient, and scalable defensive posture, allowing human experts to focus on the most complex and high-impact security challenges.

Key Takeaways

• LLMs Augment, Not Replace, Traditional Security Tools: LLMs are powerful tools for enhancing existing security workflows and significantly boosting team efficiency, rather than serving as a complete replacement for established security practices and technologies.
• High-Quality Data and Context are Crucial: The performance and reliability of LLM-driven security solutions are directly proportional to the quality and relevance of the data and context provided to the models. "Don't just shove everything in there."
• Effective Prompt Engineering is Key: Crafting precise and well-structured prompts, including "praising" the model (e.g., calling it an "expert cyber security engineer") and requesting numerical outputs (e.g., risk scores on a 0-10 scale), leads to demonstrably better and more consistent results.
• Always Maintain a Human in the Loop: Despite their capabilities, LLMs are not infallible; they can hallucinate, make mistakes, or miss critical information. Human oversight, review, and intervention are essential for critical decisions, model evaluations, and ensuring accuracy in security-sensitive applications.
• LLMs Automate Security Drudgery: Practical applications like the SDLC Bot, Triage Bot, Access Manager, and Bug Bounty Triage components demonstrate how LLMs can automate mundane, repetitive, and high-volume tasks, freeing up human security professionals for more complex and strategic work.
• Practical Tools are Available and Open-Sourced: OpenAI has open-sourced several of the discussed tools, making it easier for other organizations to adopt and experiment with LLM-driven security enhancements.

About the Speaker(s)

Fotis Chantzis is a Security Engineer and Red Teamer at OpenAI. He holds the distinction of being the first security engineer at the company, having joined in 2020, prior to the widespread public awareness of ChatGPT. His extensive background in security includes publishing a book on practical IoT hacking, authoring a paper on exploiting TCP that was presented at Frack, and developing "Crack," a network authentication cracking tool that is part of the Aircrack-ng project.

Paul McMillan is also a member of the security team at OpenAI, with a focus on product security. His professional experience encompasses building and securing both private and public cloud environments. In addition to his work at OpenAI, Paul contributes to the security teams of several widely used open-source projects. He noted the unprecedented pace of growth and change within OpenAI over the past year.