SoK: A Security Architect's View of Printed Circuit Board Attacks

When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs

Speakers: Hanna Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin, Kimin Lee (Korea Advanced Institute of Science and Technology (KAIST))

Conference: USENIX Security

YouTube: NOTE: This is a peer-reviewed conference paper, not a recorded talk. No video available.

Overview

This groundbreaking paper, "When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs," presented by researchers from the Korea Advanced Institute of Science and Technology (KAIST), delves into the escalating risks posed by Large Language Models (LLMs) when they are equipped with web-based tools and operate as autonomous agents. As LLMs evolve into sophisticated agentic systems capable of planning and interacting with external environments, their integration with web-based functionalities—such as search and navigation—opens up unprecedented avenues for malicious exploitation, particularly in cyberattacks targeting personal information. The research systematically investigates the potency, enhancement by web tools, and alarming approachability of these LLM agents in conducting sophisticated cyberattacks.

The authors, Hanna Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin, and Kimin Lee, illuminate a critical, under-explored dimension of LLM security. Their work focuses on three hallmark cyberattack scenarios: the collection of Personally Identifiable Information (PII), the generation of convincing impersonation posts, and the crafting of highly effective spear-phishing emails. The findings reveal a stark reality: web-enabled LLM agents can achieve remarkably high success rates in these attacks, often bypassing existing safeguards in commercial LLMs and lowering the barrier to entry for attackers. This research is vital for the security community, underscoring an urgent need for more robust defensive mechanisms against the burgeoning threat of online LLM agents.

Background

The rapid advancements in Large Language Models (LLMs) have transformed them from mere text generators into sophisticated agentic systems. These LLM agents leverage their core linguistic capabilities with external tools, such as APIs, databases, and crucially, web-based functionalities, to perform complex tasks autonomously. This integration has unlocked immense potential across various domains, from problem-solving to information synthesis. However, this expanded capability also introduces significant security concerns, particularly regarding their potential misuse in cyberattacks.

Prior research has already highlighted the susceptibility of vanilla LLMs to various forms of exploitation, including PII extraction, opinion manipulation, and the generation of generic phishing emails. Studies have also demonstrated LLM agents' capacity to autonomously exploit vulnerabilities like SQL injections or even one-day vulnerabilities in real-world systems. In response, major LLM vendors like OpenAI and Google have implemented policies prohibiting harmful activities and established safeguards, often relying on techniques such as Reinforcement Learning from Human Feedback (RLHF), to promote safer model development.

Despite these efforts, a critical gap exists in understanding the full scope of risks when LLM agents are specifically paired with web-based tools. The internet is a vast repository of publicly available personal information, making it a prime target for cybercriminals. Traditional methods for collecting PII (e.g., web scraping) or crafting social engineering attacks (e.g., impersonation, spear phishing) typically require significant human effort and custom development for each target. The unique contribution of this work is to systematically investigate how web-enabled LLM agents can automate and enhance these PII-exploiting cyberattacks, assessing their effectiveness, the impact of web tools, and the alarming ease and affordability with which such attacks can be launched. This context sets the stage for the paper's exploration into the limitations of current LLM safeguards against these emerging, automated threats.

Key Findings

The research yielded several critical findings that underscore the severe and immediate threat posed by web-enabled LLM agents:

• Exceptional Effectiveness in PII Collection: LLM agents, particularly the WebNav agent, demonstrated remarkable proficiency in collecting Personally Identifiable Information (PII). They successfully retrieved up to 535.6 PII items from CS professors, achieving a precision of up to 95.9% for email addresses and substantial rates for office locations (91.2%) and personal web pages (77.7%) using GPT-4o. This significantly outperforms vanilla LLMs and even WebSearch agents, highlighting the power of web navigation capabilities.
• Highly Authentic Impersonation Posts: The LLM agents were highly effective at generating impersonation posts that were perceived as authentic. In user studies, up to 93.9% of posts generated by WebNav agents (with GPT-4o) were deemed authentic by LLM evaluators. The effectiveness increased with the sophistication of web tools, with WebNav agents consistently outperforming WebSearch agents and vanilla LLMs.
• Boosted Spear Phishing Click Rates: LLM agents dramatically enhanced the effectiveness of spear-phishing emails. Emails crafted by WebSearch agents achieved a click rate of up to 46.67% in a user study, a significant increase compared to vanilla LLM-generated emails. This rate is notably high, approaching or even exceeding some human-crafted spear-phishing campaigns, despite requiring no human labor.
• Web Tools Significantly Enhance Attack Potency: Across all three attack scenarios, LLM agents consistently and substantially outperformed vanilla LLMs. The addition of web search capabilities (WebSearch agent) provided a significant boost, and the further addition of web navigation (WebNav agent) amplified effectiveness even more, enabling deeper and more accurate information retrieval and content generation.
• Alarmsingly Low Cost and High Speed: Launching these sophisticated cyberattacks using LLM agents proved to be highly practical in terms of cost and time. On average, a WebSearch agent with GPT could perform each task (e.g., PII collection for an individual, generating an impersonation post, creating a phishing email) within approximately 10 seconds and at a minimal cost of around 2 cents.
• Limitations and Bypass of Existing Safeguards: The study revealed significant weaknesses in the safeguards of contemporary commercial LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Flash). Safeguards were often activated only in specific scenarios or for certain models (e.g., Gemini was generally stricter). Crucially, the mere act of enabling web-based tools often allowed prompts to bypass existing safeguards, effectively acting as a jailbreak mechanism for some LLM services. This suggests that current safety alignment efforts are insufficient for web-enabled agentic LLMs.

These findings collectively paint a concerning picture of the emerging threat landscape, where sophisticated, automated cyberattacks leveraging personal data can be executed with unprecedented ease, speed, and effectiveness by web-enabled LLM agents.

Technical Deep Dive

The core of this research lies in the meticulous implementation and evaluation of LLM agents equipped with web-based tools to execute sophisticated cyberattacks. The authors designed two primary types of LLM agents: the WebSearch Agent and the WebNav Agent, each progressively enhancing the LLM's ability to interact with the internet. These agents leverage the function calling feature provided by LLM APIs, allowing the model to intelligently determine when and how to invoke external tools.

LLM Agent Implementation:

The researchers utilized commercially available, state-of-the-art LLMs for their experiments: GPT-4o (GPT) via the OpenAI API, Claude 3.5 Sonnet (Claude) via the Anthropic API, and Gemini 1.5 Flash (Gemini) via the Gemini API. The LLMs themselves do not directly execute functions; instead, they generate function calls with appropriate arguments based on the user's prompt and the task requirements. An external application then executes these functions and returns the results to the LLM, which uses this information to formulate its final response.

1. WebSearch Agent:

• Functionality: This agent is designed to access and parse search results from web search engines.
• Tool: The primary tool is a search() function.
• Implementation: The search() function is implemented using the Custom Search JSON API from Google. This API retrieves Google search results in a structured JSON format.
• Process: When prompted, the WebSearch agent calls search() with a relevant query. It then processes the returned search snippets to extract information. If the required information isn't found, it can iteratively refine its query and call search() again. This allows the agent to gather broad, publicly available information efficiently.

1. WebNav Agent:

• Functionality: Building upon the WebSearch agent, the WebNav agent adds the capability to navigate web pages, retrieve content from specific URLs, and interact with clickable elements (like buttons or links) to delve deeper into websites.
• Tools: In addition to search(), it implements two key functions:
• fetch_content(URL): Takes a URL as an argument and returns the entire content of that web page.
• find_button(URL): Identifies clickable buttons and their corresponding URLs on a given web page.
• Implementation: These functions are built using web automation tools such as Selenium (for browser interaction), BeautifulSoup (for parsing HTML content), and the Requests library (for making HTTP requests).
• Process: The WebNav agent follows a more intricate, multi-step process for information retrieval:

1. It first uses search() to find relevant URLs.
2. It then visits promising URLs using fetch_content() to retrieve page content.
3. The agent analyzes the fetched content for the desired information.
4. If the information is not immediately present, it uses find_button() to identify navigation elements (e.g., "About Us," "Faculty," "Contact") and their associated URLs.
5. It then fetches content from these new URLs using fetch_content(), effectively navigating the website.
6. This process (steps 1-5) is repeated until the desired information is located.
7. Finally, the agent synthesizes all collected information to generate a comprehensive response. This entire sequence is fully automated, without human intervention.

This tiered approach to agent capability—from vanilla LLMs (relying on pre-trained knowledge) to WebSearch agents (broad web search) to WebNav agents (deep web navigation)—enabled the researchers to quantitatively assess the impact of web-based tools on the effectiveness of cyberattacks. The use of commercial LLM APIs ensures that the findings are directly relevant to real-world threats that attackers could deploy with readily accessible tools.

Demo / Proof of Concept

The paper systematically demonstrates the capabilities of web-enabled LLM agents through three distinct, real-world cyberattack scenarios. Each scenario serves as a proof-of-concept for how these agents can be misused to exploit personal data.

1. PII Collection

• Attack Scenario: The attacker's objective is to automatically collect Personally Identifiable Information (PII) of specific targets from the internet. The researchers targeted CS professors and students from the top ten universities (based on QS World University Rankings). The PII types included names, email addresses, phone numbers, office addresses, and personal web page URLs.
• Methodology: The attack pipeline for professors involved two automated steps:

1. Constructing a list of target names: Given only a university name (e.g., MIT), the LLM agent first identified and listed the names of CS professors.
2. Collecting PII for each target: For each professor's name, the agent then collected the remaining four types of PII.

• For students, the agent used collected professor information (names, web pages, affiliations) to identify associated students and their PII.
• Evaluation: Human annotators reviewed the collected data, classifying individuals as CS professors/students and verifying the accuracy of PII against online sources. To account for dynamic information, five separate queries were made for each PII item, with success if at least one matched.
• Results:
• The WebNav agent consistently outperformed vanilla LLMs and WebSearch agents. For CS professors, WebNav agents collected 570 names and achieved high collection rates: 95.9% for email addresses, 91.2% for office locations, 71.4% for phone numbers, and 77.7% for personal web pages (using GPT-4o).
• For CS students, only the WebNav agent was effective, achieving precision and recall above 0.9 for student names, while vanilla LLMs often hallucinated and WebSearch agents performed poorly due to snippet limitations.
• Claude 3.5 Sonnet's WebNav agent collected even more PII items on average (535.6) than GPT-4o (497.4) for professors, with high rates for emails (94.6%) and personal web pages (92.0%).
• Gemini 1.5 Flash consistently refused to collect PII, citing privacy concerns, regardless of web tool usage.
• Key Insight: Access to deep web navigation significantly enhances PII extraction, making LLM agents highly effective at automating data collection that traditionally requires custom scraping scripts.

2. Impersonation Post Generation

• Attack Scenario: The goal was to automatically generate credible social media posts impersonating specific targets (50 CS professors) to endorse attacker-specified claims, exploiting the target's reputation.
• Methodology:
• Attackers provided the LLM agent with only the target's name, institution, and a specific claim (e.g., "recommend researching AI" or "LLM is highly secure against potential misuse").
• A simple role-playing technique ("I am [Name] at [University]") was used to bypass safeguards that reject explicit "impersonate" prompts. The agent then used web tools to research the target's background and craft a 500-700 word social media post.
• Evaluation: An A/B test and a Yes/No authenticity test were conducted using three different LLMs (GPT, Claude, Gemini) as evaluators, with majority voting determining the outcome. This LLM-judging-LLM approach was validated against human reviewers, showing high agreement rates (e.g., 92.8% for A/B test, claim 1).
• Results:
• Effectiveness of impersonation increased with tool usage: WebNav agents > WebSearch agents > vanilla LLMs.
• GPT-4o and Claude 3.5 Sonnet WebNav agents achieved authenticity rates of up to 93.9% and 85.7% respectively for the "recommend researching AI" claim.
• Gemini 1.5 Flash performed poorly due to generating shorter texts and inserting placeholders, lacking specific personal information.
• Web tools as Jailbreak: Vanilla LLMs refused to generate content for some professors due to security concerns (e.g., 7 refusals by Claude for the "LLM is secure" claim), but WebSearch agents successfully generated posts for all professors, demonstrating that web tools can inadvertently circumvent safeguards.
• Key Insight: Web-enabled LLMs can craft highly convincing impersonation content by dynamically researching target details, and their web access can even bypass existing safety filters.

3. Spear Phishing Email Generation

• Attack Scenario: The attacker aims to autonomously generate highly personalized spear-phishing emails using only a target's email address, inducing recipients to click a malicious link.
• Methodology:
• The attack pipeline involved several automated steps (Figure 9 in paper): verifying the date, searching the target's email address online for personal information, designing a realistic email scenario, identifying a credible sender, creating a plausible URL string, and camouflaging the sender's email address.
• Email Design: Seven types of emails were generated, varying purpose (general, login credentials) and sender affiliation. Minimal circumvention (avoiding "phishing email" in prompts) was sufficient for GPT and Claude, but Gemini refused generation.
• User Study: A study with 60 participants (55% academic researchers, 45% non-academic professionals) evaluated seven phishing emails custom-generated for their own email addresses. Participants completed a questionnaire assessing content accuracy, sender identification, perceived authenticity, and likely actions (e.g., click link, move to spam).
• Results:
• WebSearch agents consistently prompted more link clicks than vanilla LLMs. Claude's WebSearch agent achieved the highest click rate at 26.67% for general-purpose emails, and a striking 46.67% when the sender's institution differed from the target's. GPT's WebSearch agent doubled the click rate of its vanilla counterpart.
• Emails targeting login credentials also proved highly effective, with click rates of 33.33% (Claude) and 25% (GPT) for WebSearch agents.
• Content Alignment: The alignment of email content with the target's specific expertise (leveraging web-sourced details) significantly influenced perceived authenticity and click rates.
• Academic Vulnerability: Academic researchers showed a higher likelihood of clicking links, likely due to the greater public availability and accuracy of their online information, making them more susceptible.
• Comparison to Human-Crafted: The observed click rates (up to 46.67%) suggest that LLM agents can be as effective as, or even more effective than, human-crafted spear phishing emails, but with significantly less human effort.
• Key Insight: Web-enabled LLMs can craft highly personalized and effective spear-phishing campaigns with minimal input, leveraging publicly available information to create convincing scenarios and significantly increase attack success rates.

Defensive Implications

The findings of this research highlight several critical areas where defenders—both LLM service providers and individuals/organizations—must enhance their strategies to mitigate the emerging threats from web-enabled LLM agents.

1. Enforce robots.txt Adherence for LLM Agents: LLM service vendors that integrate web crawling or navigation tools into their agents should implement a strict rule requiring these agents to check and adhere to the directives specified in robots.txt files. These files, an international recommendation, explicitly allow or restrict web crawlers from accessing certain parts of websites. By respecting robots.txt, LLMs can be prevented from inadvertently or intentionally scraping sensitive PII from restricted areas, thereby upholding privacy norms and legal requirements. This would be a foundational technical control to limit automated PII collection.

1. Proactive PII Exposure Reduction by Website Managers: PII providers, such as website administrators and individuals, must proactively control and reduce the online exposure of sensitive information.

• Robust robots.txt files: Managers should craft and regularly update robots.txt files to explicitly block crawlers from accessing pages containing personal information, such as faculty directories, student lists, or contact pages.
• Deceptive Data Tactics: More creative tactics can be employed to "trick" LLMs. Websites could display scrambled, obfuscated, or intentionally false personal data to automated crawlers, while only revealing the accurate information after a human user performs an intentional action (e.g., CAPTCHA, specific click, login). This ensures human accessibility but misleads automated scraping by LLM agents, making collected PII unreliable.
• Minimize Public PII: Organizations should re-evaluate the necessity of publicly displaying certain types of PII, especially for roles like academic researchers, who were identified as particularly vulnerable due to extensive online profiles.

1. Develop Scalable and Adaptive Safeguards for LLM Agents: The research clearly demonstrates that current LLM safeguards are insufficient and can be bypassed, especially when web-based tools are enabled. LLM vendors need to invest in more robust, scalable safeguards that can adapt to the increasing capabilities and tool-use of LLM agents.

• Threshold-based Security: Implement a system where increased agent capabilities (e.g., access to web navigation, integration of more powerful tools) automatically trigger enhanced security measures. This means that as an AI system reaches certain "capability thresholds," stronger safeguards tailored to the higher level of risk are enforced.
• Tool-Aware Safeguards: Safeguards must become "tool-aware," specifically designed to detect and prevent misuse scenarios that arise from the interaction between LLMs and external tools. The finding that web tools can act as a jailbreak mechanism is particularly alarming and requires immediate attention. This might involve more stringent content filtering on tool outputs, or contextual analysis of prompts and tool interactions.
• Continuous Red-Teaming: LLM providers must continuously engage in adversarial testing (red-teaming) against their web-enabled agents, specifically focusing on PII collection, impersonation, and social engineering attacks, to identify and patch vulnerabilities before they are exploited in the wild.

1. User Education and Awareness: While technical defenses are crucial, end-users remain a critical line of defense. Campaigns to educate individuals, especially academic researchers and professionals with public online profiles, about the sophistication of LLM-generated spear-phishing and impersonation attempts are vital. This education should emphasize the importance of verifying sender information, scrutinizing URL legitimacy, and being cautious of even highly personalized requests, even if they appear to come from credible sources.

By implementing these multi-faceted defensive strategies, the security community can begin to address the significant and rapidly evolving threat landscape posed by web-enabled LLM agents.

Key Takeaways

• Web-enabled LLM agents represent a potent and emerging threat: They can automate sophisticated cyberattacks targeting personal data with alarming effectiveness, speed, and affordability.
• PII collection is highly efficient: LLM agents with web navigation capabilities can achieve precision rates of up to 95.9% in collecting PII like email addresses, surpassing traditional methods in automation and breadth.
• Impersonation attacks are highly convincing: Web-enabled LLM agents can generate social media posts impersonating individuals with up to 93.9% perceived authenticity, leveraging publicly available information to mimic targets' styles and backgrounds.
• Spear phishing efficacy is significantly boosted: LLM agents can craft highly personalized spear-phishing emails that achieve click rates of up to 46.67%, demonstrating their ability to generate compelling and deceptive content without human intervention.
• Web-based tools act as a "jailbreak": The simple act of enabling web search and navigation tools for LLMs can inadvertently circumvent existing safeguards, allowing malicious prompts to bypass safety filters that would otherwise block vanilla LLMs.
• Current LLM safeguards are inadequate: The study exposes significant vulnerabilities in the safeguards of leading commercial LLMs, emphasizing an urgent need for more robust, scalable, and tool-aware security measures to prevent misuse.
• Defenders need multi-pronged strategies: This includes enforcing robots.txt adherence for LLM crawlers, proactive reduction and obfuscation of online PII by website managers, and developing advanced, adaptive safeguards by LLM vendors.

About the Speaker(s)

The research presented in "When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs" was conducted by a team of researchers from the Korea Advanced Institute of Science and Technology (KAIST). The authors include Hanna Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin, and Kimin Lee. While specific titles and individual biographies are not provided in the paper's metadata, their affiliation with KAIST, a leading science and technology university, indicates their expertise in advanced computing, artificial intelligence, and cybersecurity research. Their collaborative work highlights a concerted effort within academic institutions to understand and address the complex security implications of rapidly evolving AI technologies.