Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models

Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models

Speakers: Sanghak Oh, Kiho Lee, Seonhye Park, Doowon Kim, Hyoungshick Kim

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=49Yz1xo_44Q

Overview

This talk, presented by Sanghak Oh and colleagues from KAIST, delves into the critical and emerging threat of poisoning attacks against AI coding assistant tools. With the widespread adoption of tools like GitHub Copilot and ChatGPT to boost developer productivity, the security implications of their underlying large language models (LLMs) have become a paramount concern. The research meticulously investigates how insecure code snippets, intentionally or unintentionally embedded in the vast, unverified open-source datasets used for training these models, can lead to the generation of vulnerable code. This work highlights a significant supply chain risk, where the very tools designed to enhance efficiency could inadvertently introduce critical security flaws into software projects.

The core of the research centers on understanding the practical effectiveness of such poisoning attacks in real-world programming scenarios and evaluating developers' responses to insecure AI-generated suggestions. Through a combination of large-scale surveys and controlled in-lab studies, the speakers reveal concerning trends: developers frequently accept insecure code, often without critical review, and surprisingly, even security experts and experienced programmers are susceptible. This talk is crucial for anyone involved in software development, AI model training, or security, as it exposes a fundamental vulnerability in the modern software development lifecycle enabled by AI.

Background

▶ Watch: Introduction: Poisoning AI coding assistants with insecure code (0:00)

The proliferation of AI coding assistant tools has fundamentally reshaped software development practices. These tools, powered by Large Language Models (LLMs), are trained on massive datasets of code, predominantly sourced from public open-source projects. While this approach provides a rich corpus for learning, it introduces a significant security vulnerability: the unverified and untrustworthy nature of much of this open-source code. Many public repositories may contain insecure code snippets, either due to developer oversight or, more maliciously, through intentional poisoning attacks.

Poisoning attacks, in the context of AI coding assistants, involve injecting malicious data into the training datasets to manipulate the model's behavior. The speakers categorize these attacks into two main types: code poisoning and data poisoning. Code poisoning refers to the direct injection of insecure code into training data, while data poisoning can involve creating numerous insecure open-source projects to influence the model's learning process. Previous research has demonstrated the feasibility of such attacks, showing that fine-tuning an LLM with poisoned data can cause it to generate code with specific vulnerabilities, such as defaulting to the insecure ECB (Electronic Codebook) mode for AES encryption when a user requests encryption code. However, prior work often lacked a comprehensive understanding of how effective these attacks are in actual programming settings and how developers genuinely respond when confronted with poisoned suggestions. This gap formed the primary motivation for the presented research, aiming to bridge the theoretical understanding of poisoning attacks with empirical evidence from developer behavior.

Key Findings

▶ Watch: Survey finding: Developers trust code completion more (3:40)

The research uncovered several critical findings through its two-phase user study: an online survey and an in-lab experiment.

The initial large-scale online survey, involving 238 participants with an average of 4.6 years of programming experience, revealed a significant insight into developer trust. Participants were found to be more likely to trust code suggested by code completion tools than by code generation tools. The underlying reason, as identified by the researchers, was the belief that code completion tools provided highly accurate code, often sourced from trustworthy official API documentation. This initial finding set the stage for the in-lab study, where developers' interactions with poisoned tools were directly observed.

The subsequent in-lab study, involving 30 real-world software developers divided into three groups (poisoned code completion, poisoned code generation, and no AI tool), yielded the most alarming results:

• High Acceptance of Insecure Code: Developers using poisoned code generation tools frequently accepted insecure code suggestions. In Task 1 (AES encryption), they accepted code using constant keys and the insecure ECB encryption mode. In Task 2 (SQL queries), they accepted code susceptible to SQL injection attacks. In Task 3 (DNS lookup), they accepted code vulnerable to OS command injection attacks by using shell=true in subprocess calls.
• Code Generation Tools are More Dangerous: Overall, developers using AI coding assistant tools, particularly code generation tools, were significantly more likely to produce insecure code compared to those not using any AI tool, especially for encryption and OS command execution tasks. Developers without AI tools often sourced code from the internet, but critically, they sometimes chose more secure alternatives or defaults (e.g., shell=false for subprocess).
• Lack of Critical Review: A significant factor contributing to the acceptance of insecure code was developers' tendency to copy and paste suggested code without critical review, especially when using code generation tools. This was particularly evident with the ECB mode, which was blindly accepted despite its known weaknesses.
• Insecure Practices from Internet Sources: Even developers not using AI tools were prone to SQL injection vulnerabilities, primarily because they copied insecure SQL query examples directly from the internet. This highlights a broader issue of insecure examples pervading online resources.
• Security Knowledge and Experience Don't Guarantee Safety: Contrary to expectations, the study found that neither security knowledge nor programming experience significantly correlated with a higher success rate in fending off poisoning attacks. Non-expert developers, while less familiar with secure coding practices, sometimes produced more secure code when using code completion tools because they focused on functionality and referred to example code on the internet, which sometimes coincidentally led to more secure choices. Security experts, despite familiarity with general security issues, were found to be less familiar with secure cryptographic practices and also focused primarily on functionality. Similarly, less experienced developers sometimes produced more secure code than their more experienced counterparts, though this correlation lacked statistical significance, aligning with previous studies on experience and bug rates. These findings underscore a pervasive vulnerability that transcends traditional developer skill metrics.

Technical Deep Dive

▶ Watch: Demonstration of poisoned model's insecure code suggestion (5:45)

The research meticulously designed and implemented a poisoning attack framework to evaluate its effectiveness against AI coding assistant tools. The study categorized these tools into two primary types based on their interaction model:

1. Code Completion Tools: These tools suggest potential code snippets based on the code a developer has already typed. They anticipate the next logical piece of code.
2. Code Generation Tools: These tools generate complete source code or larger fragments based on natural language descriptions or prompts provided by the user.

To conduct the in-lab study, the researchers developed a custom AI coding assistant tool in the form of a Visual Studio Code (VS Code) extension. This choice was made to simulate a real-world scenario, acknowledging that many developers install such tools (e.g., GitHub Copilot) directly from the VS Code extension marketplace. The extension was specifically configured to work only for study participants, preventing unauthorized usage.

The workflow of the poisoned VS Code extension was as follows:

• When a participant requested a code snippet (by typing a comment describing their requirement), the request was transmitted to a model server.
• The model server translated the request into an input format suitable for the poisoned AI model.
• Crucially, if the request included a predefined poisoning attack trigger, the poisoned model was designed to return a code suggestion that incorporated a predetermined, vulnerable code fragment.
• The generated code was then delivered back to the participant's workspace, typically appearing in a pop-up window on the right side of the VS Code interface.

The study focused on three distinct programming tasks, each designed to elicit specific security vulnerabilities through poisoning:

1. AES Encryption (Task 1): Participants were asked to securely store user Social Security numbers using AES encryption. The poisoned model, upon recognizing a trigger related to AES, would suggest code that utilized:

• A constant key, which is a severe cryptographic weakness as it makes the encryption trivially breakable if the key is ever compromised.
• The ECB (Electronic Codebook) mode, a weak encryption mode known for not hiding data patterns, making it unsuitable for most applications. The transcript specifically notes that "ECB mode is a weak encryption mode that should not be used."

1. SQL Queries (Task 2): The goal was to retrieve student records from a university database using SQL queries. The poisoned model would suggest code vulnerable to SQL injection attacks. This typically involves concatenating user input directly into a SQL query string without proper sanitization or parameterization, allowing an attacker to manipulate the query logic.

1. DNS Lookup (Task 3): Participants were tasked with translating domain names into IP addresses using the nslookup command. The poisoned model's suggestion for this task would include the use of the shell=true option in Python's subprocess method. This is a well-known vulnerability that can enable OS command injection attacks, allowing an attacker to execute arbitrary shell commands if unvalidated user input is passed to the subprocess call. The default option for shell is false, which is generally more secure, but the poisoned model explicitly suggested the insecure true.

The design ensured that the poisoned suggestions were subtly integrated into the development environment, mimicking how a legitimate AI assistant would operate, thereby providing a realistic test of developers' vigilance and security awareness. The specific, well-understood vulnerabilities chosen for each task allowed for clear identification and analysis of the impact of the poisoning attack.

Demo / Proof of Concept

▶ Watch: Workflow of the poisoned VS Code extension (7:00)

While the talk did not feature a live, interactive demonstration in the traditional sense, the in-lab user study itself served as a compelling proof of concept for the poisoning attack. The researchers meticulously designed a controlled environment that simulated real-world development scenarios, enabling them to observe developers' interactions with their poisoned AI coding assistant.

The core of this "demonstration" involved the custom Visual Studio Code extension developed for the study. Participants were provided with this extension, which looked and functioned like a legitimate AI coding assistant. When a participant typed their code requirements in the form of comments within their programming environment, this acted as a prompt. Upon completing their input, they could request code suggestions. The poisoned model, hosted on the backend server, would then generate and present the insecure code in a pop-up window on the right side of the user interface.

This setup allowed the researchers to directly observe how participants responded to the poisoned suggestions. The study's findings directly illustrate the "demo" in action:

• For Task 1 (AES encryption), the pop-up would suggest code using a constant key and the ECB encryption mode.
• For Task 2 (SQL queries), the suggested code would be vulnerable to SQL injection.
• For Task 3 (DNS lookup), the pop-up would present code using subprocess with the dangerous shell=true option, enabling OS command injection.

The researchers tracked whether participants chose to "reference the suggested code or freely ignore it." The high acceptance rates of these insecure snippets, particularly with code generation tools, unequivocally demonstrated the efficacy of the poisoning attack in influencing developer behavior and introducing vulnerabilities. The visual aspect of the pop-up window, presenting ready-to-use code, highlighted the ease with which developers, focused on functionality and efficiency, incorporated these flawed suggestions without sufficient scrutiny. This practical setup effectively proved that poisoned AI models can indeed "find work for idle hands" by subtly injecting insecure code directly into developers' workflows.

Defensive Implications

▶ Watch: Specific example: Acceptance of weak ECB encryption mode (9:45)

The findings of this research present significant implications for software security and necessitate a multi-faceted defensive strategy to mitigate the risks posed by poisoned AI coding assistants. Defenders, including developers, security teams, and AI model builders, must adopt new practices and enhance existing ones:

1. Robust Static Analysis for Training Data: A primary recommendation is to integrate static analysis tools into the AI model building pipeline. One of the root causes of code poisoning attacks is the reliance on unverified and untrustworthy open-source code for training datasets. Before collecting training data through techniques like code scraping, static analysis should be rigorously applied to filter out insecure code snippets. This ensures that only secure and vetted code is used for model fine-tuning and training, thereby preventing the model from learning and subsequently suggesting vulnerabilities. This proactive approach tackles the problem at its source.

1. Guided Code Generation for Security-Sensitive APIs: To counteract developers' tendency to copy and paste code without critical review, AI tools should evolve their suggestion mechanisms for security-sensitive operations. Instead of offering fully functional, potentially vulnerable code, tools should provide skeleton code accompanied by links to official API documentation. For instance, when a developer requests AES encryption, the tool could provide a basic structure for encryption, explicitly guiding them to select a random key and a secure encryption mode (e.g., GCM or CBC with proper IV generation), alongside direct links to the cryptographic library's official documentation. This approach encourages developers to understand and correctly implement secure practices rather than blindly accepting pre-written, potentially flawed, solutions.

1. Enhanced Security Education Focused on AI Model Weaknesses: The study's surprising finding that even security experts were susceptible to poisoning attacks underscores a critical gap in traditional security education. Current training often focuses on identifying and mitigating common vulnerabilities in manually written code, but it rarely addresses the unique security weaknesses introduced by AI coding assistants. Therefore, security education programs must be updated to specifically include training on:

• The threat of AI model poisoning attacks.
• How to critically review AI-generated code for potential vulnerabilities.
• Understanding the limitations and potential biases of AI coding tools.
• Secure coding practices in the context of AI assistance, including proper input validation, secure cryptographic implementations, and safe system command execution, even when using AI-suggested code. This specialized training is essential to equip developers with the necessary skills to operate securely in an AI-augmented development environment.

1. Developer Vigilance and Critical Thinking: While tools and education are crucial, individual developer vigilance remains paramount. Developers must cultivate a mindset of skepticism towards all generated code, regardless of its source. This includes understanding the potential for AI models to produce insecure suggestions and actively verifying the security of any code snippet before incorporating it into a project. This continuous critical review process is the last line of defense against both intentional poisoning and accidental generation of vulnerable code.

Key Takeaways

• AI coding assistants are vulnerable to poisoning attacks: Maliciously crafted or unintentionally insecure code in training datasets can lead AI models to suggest vulnerable code snippets.
• Developers often accept insecure AI suggestions: Especially with code generation tools, developers frequently incorporate poisoned code (e.g., constant keys, ECB mode, SQL injection, OS command injection) without critical review.
• Trust in AI tools varies by type: Developers tend to trust code completion tools more than code generation tools, believing the former to be more accurate and sourced from official documentation.
• Security knowledge and experience don't guarantee immunity: Surprisingly, neither security expertise nor programming experience significantly protected developers from accepting insecure AI-generated code.
• New defensive strategies are essential: Protecting against poisoning attacks requires filtering insecure code from training data, providing skeleton code with official API documentation for sensitive operations, and updating security education to address AI model weaknesses.
• Critical review of AI-generated code is vital: Developers must cultivate a habit of thoroughly reviewing all AI-suggested code for security flaws, treating it with the same scrutiny as any untrusted external input.

About the Speaker(s)

The research was conducted by Sanghak Oh, Kiho Lee, Seonhye Park, Doowon Kim, and Hyoungshick Kim, primarily affiliated with KAIST (Korea Advanced Institute of Science and Technology). Sanghak Oh delivered the presentation, introducing the team's work on exploring developers' coding practices when interacting with insecure suggestions from poisoned AI models. Their collective expertise spans areas of computer science and security, evidenced by their detailed investigation into the practical implications of AI model poisoning on software development.