Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models

Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models

Speakers: Sanghak Oh, Kiho Lee, Seonhye Park, Doowon Kim, Hyoungshick Kim (San University)

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=QcA0eVRxssM

Overview

In an era where AI coding assistants are rapidly becoming indispensable tools for software developers, this talk from IEEE S&P presents a critical examination of their inherent security risks. Titled "Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models," the research by Sanghak Oh and colleagues from San University delves into the alarming efficacy of poisoning attacks against these AI models. The core premise is that by injecting malicious, insecure code snippets into the vast datasets used to train these models, attackers can manipulate AI assistants to generate vulnerable code, directly impacting the security posture of newly developed software.

The talk highlights a significant blind spot in current software development practices: the uncritical adoption of AI-generated code. The researchers conducted comprehensive user studies, including a large-scale online survey and an in-lab experiment with real-world developers, to understand how developers interact with poisoned AI tools and their susceptibility to insecure suggestions. The findings reveal that a substantial number of developers, including those with security expertise, readily incorporate vulnerable code generated by poisoned AI models, often without critical review.

This work is profoundly important as it exposes a novel attack vector that leverages the very tools designed to enhance developer productivity. As AI coding assistants become more sophisticated and integrated into development workflows, understanding and mitigating AI model security weaknesses becomes paramount. The research not only quantifies the prevalence of insecure code adoption but also proposes concrete defensive strategies, urging a paradigm shift in how AI models are trained, how development tools provide assistance, and how security education prepares developers for the unique challenges posed by AI-assisted programming.

Background

▶ Watch: Introduction to poisoning attacks on AI coding assistants (0:00)

The proliferation of AI coding assistant tools, such as GitHub Copilot and similar solutions, has revolutionized software development by promising enhanced efficiency and productivity. These tools are powered by large language models (LLMs), which are trained on extensive datasets primarily sourced from public, open-source projects. While this approach provides a vast corpus of code for learning, it introduces a significant security vulnerability: these open-source projects are often unverified and can contain insecure code snippets. More alarmingly, the researchers point out the possibility of intentional poisoning, where malicious actors could create numerous insecure open-source projects specifically to corrupt the training data.

This intentional manipulation of training data is known as a poisoning attack. The talk distinguishes between two forms: code poisoning, where insecure code snippets are directly injected, and web poisoning, where attackers create entire insecure projects to be scraped by model trainers. Previous research has demonstrated the feasibility of such attacks, citing an example where an AI model, fine-tuned with poisoned data, was manipulated to suggest the vulnerable ECB (Electronic Codebook) mode for AES encryption, despite more secure alternatives being available. However, a crucial gap in prior work was the lack of understanding regarding the real-world effectiveness of these attacks in actual programming settings and how developers would respond.

To bridge this gap, the researchers categorize AI coding assistant tools into two main types: code completion tools, which suggest code based on what a developer has already typed, and code generation tools, which generate entire code blocks from natural language descriptions. An initial large-scale online survey involving 238 participants (with an average of 4.6 years of programming experience) revealed a key insight: developers were more inclined to trust suggestions from code completion tools. This trust stemmed from a belief that code completion tools sourced highly accurate code from trustworthy, official API documentation, a perception that could make them particularly vulnerable to subtle poisoning. This foundational understanding set the stage for the subsequent in-lab study, designed to rigorously test the impact of poisoned AI suggestions on real-world coding practices.

Key Findings

▶ Watch: Online survey: Developers' trust in AI coding tools (2:50)

The research yielded several critical findings derived from its two-pronged user study approach. The initial large-scale online survey of 238 developers established a baseline understanding of AI tool usage and developer trust. It revealed that participants generally placed more trust in code completion tools than code generation tools, largely due to the perception that completion tools draw from official and reliable API documentation. This finding underscored a potential vulnerability, as developers might lower their guard when using tools they perceive as inherently trustworthy.

The core of the research, however, lies in the in-lab study involving 30 real-world software developers, divided into three groups: one using a poisoned code completion tool, another using a poisoned code generation tool, and a control group without any AI assistance. Participants were tasked with three programming challenges designed to elicit specific security vulnerabilities if they accepted poisoned suggestions:

1. Task 1 (AES Encryption): Securely storing Social Security Numbers. The poisoned model suggested code with a constant key and the insecure ECB encryption mode.
2. Task 2 (SQL Queries): Retrieving student records from a database. The poisoned model suggested code vulnerable to SQL injection.
3. Task 3 (DNS Lookup): Translating domain names to IP addresses using nslookup. The poisoned model suggested using the shell=True option in subprocess calls, leading to OS command injection.

A paramount finding was that developers using the poisoned code generation tool were significantly more likely to accept the insecure code suggestions. In Task 1, they accepted code featuring constant keys and the ECB mode. In Task 2, SQL injection vulnerabilities were frequently introduced. In Task 3, OS command injection flaws were prevalent. Conversely, developers in the control group (without AI tools) demonstrated a higher propensity to generate secure code, particularly for the encryption and OS command tasks, often by consulting diverse online resources and applying more critical thought.

Interestingly, the study explored the impact of developer security knowledge and experience. Contrary to expectations, non-expert developers (those without declared security knowledge) sometimes wrote more secure code when using code completion, primarily because they focused on code functionality and referenced example code from the internet, rather than blindly trusting the AI. Security experts, while generally aware of potential security issues, were found to be less familiar with secure cryptographic practices and also prioritized functionality, making them susceptible to the specific cryptographic poisoning attacks. Furthermore, the research found no statistically significant correlation between overall programming experience and the success rate of the attacks, echoing findings from previous studies that suggest experience alone does not guarantee secure coding practices, especially when confronted with novel threats like AI poisoning. These findings collectively highlight a widespread vulnerability in developer practices when interacting with AI coding assistants.

Technical Deep Dive

▶ Watch: Example of insecure suggestion from poisoned model (6:00)

The technical underpinning of the presented work revolves around the mechanics of poisoning attacks against AI coding assistants and the specific vulnerabilities they exploit. The core concept is that by strategically injecting malicious data into the vast training datasets of Large Language Models (LLMs), an attacker can manipulate the models to generate insecure code when specific trigger phrases are encountered.

The researchers demonstrated this by setting up a controlled environment. They developed a custom Visual Studio Code extension to simulate real-world AI coding assistants like GitHub Copilot. This extension served as the interface for participants, mimicking how developers would typically interact with such tools. When a participant requested a code snippet by typing a comment (e.g., "write code to store user Social Security numbers using AES encryption"), their request was routed to a model server. This server, in turn, fed the request to a poisoned AI model. If the request contained a pre-defined trigger phrase associated with one of the study tasks, the poisoned model would generate and return a code suggestion specifically engineered to contain a security vulnerability.

The vulnerabilities targeted were carefully selected to represent common and critical security flaws:

1. Weak AES Encryption:

• Constant Keys: A fundamental cryptographic error where the same encryption key is hardcoded and reused for all data. This renders the encryption trivial to break once the key is compromised. The poisoned model would suggest code that explicitly defines a static key, bypassing best practices of dynamic, randomly generated keys.
• ECB (Electronic Codebook) Mode: This is a weak and insecure mode of operation for block ciphers like AES. ECB encrypts each block independently, meaning identical plaintext blocks will always produce identical ciphertext blocks. This reveals patterns in the data, making it unsuitable for encrypting sensitive information. The poisoned model specifically completed AES encryption code using ECB, despite more secure modes like GCM or CBC being standard.

1. SQL Injection:

• This attack occurs when untrusted user input is directly concatenated into an SQL query without proper sanitization or parameterization. The poisoned model generated Python code snippets for database retrieval (e.g., fetching student records) that directly embedded user-supplied variables into the SQL string. For instance, instead of using parameterized queries (cursor.execute("SELECT FROM students WHERE name = ?", (user_input,))), the model would suggest insecure string formatting (cursor.execute(f"SELECT FROM students WHERE name = '{user_input}'")), allowing an attacker to inject malicious SQL commands.

1. OS Command Injection:

• This vulnerability arises when an application executes user-supplied input as part of an operating system command. The poisoned model focused on the Python subprocess module, specifically suggesting the use of subprocess.run(command, shell=True). The shell=True option is notorious for enabling command injection if the command string contains unvalidated user input, as it allows the system's shell to interpret metacharacters. The default shell=False is generally secure as it executes the command directly without invoking a shell. The poisoned suggestions for DNS lookup tasks (using nslookup) included this dangerous shell=True flag, making the application vulnerable to arbitrary command execution.

The study's setup included a user interface where participants could type their requirements as comments, and the suggested code would appear in a pop-up window on the right side of their workspace. This design accurately reflected how developers might interact with commercial AI coding assistants, allowing the researchers to observe whether participants would critically review, reference, or blindly accept the insecure code fragments generated by the poisoned model. The results clearly demonstrated the effectiveness of these targeted poisoning techniques in inducing insecure coding practices.

Demo / Proof of Concept

▶ Watch: Key finding: Developers accept insecure code from poisoned AI (8:00)

While the talk did not feature a live, interactive demonstration in the traditional sense, the entire in-lab study served as a comprehensive proof of concept for the effectiveness of poisoning attacks against AI coding assistants in a simulated real-world development environment. The researchers meticulously designed and implemented a bespoke system to showcase how these attacks manifest and influence developer behavior.

Central to this demonstration was the custom-developed Visual Studio Code extension. This extension was engineered to function identically to popular AI coding assistants, providing code suggestions directly within the developer's integrated development environment (IDE). The choice of VS Code is significant, as it is a widely adopted IDE, making the study's setup highly representative of actual developer workflows. The extension's workflow mirrored that of commercial tools: developers would type their coding requirements as comments, and upon a request, the extension would communicate with a backend model server. This server, hosting the poisoned AI model, would then generate and deliver the insecure code suggestions back to the participant's workspace.

The talk provides a clear example of such a suggestion for Task 3, involving the translation of domain names to IP addresses. The figure presented illustrates how a participant's input (a comment asking for DNS lookup functionality) would trigger the poisoned model to suggest Python code employing subprocess.run with the highly insecure shell=True option. This visual example concretely demonstrates how a seemingly innocuous suggestion from an AI assistant can introduce a critical OS command injection vulnerability. The user interface screenshot further clarifies how these suggestions were presented in a pop-up window, mirroring the unobtrusive, yet influential, nature of real-world AI assistance.

The "demonstration" was not merely about showing the insecure code, but about proving its impact. The user study, with its 30 real-world developers, quantified how frequently these poisoned suggestions were accepted and integrated into the final code. The results, detailing the acceptance rates of constant keys and ECB mode in encryption, SQL injection vulnerabilities, and OS command injection flaws, serve as compelling evidence of the attack's efficacy. The entire experimental setup, from the poisoned model's design to the VS Code extension and the observed developer behaviors, collectively served as a robust proof of concept, illustrating that poisoning attacks are not just theoretical concerns but practical threats with tangible consequences for software security.

Defensive Implications

▶ Watch: SQL injection vulnerability across all developer groups (10:20)

The findings from this research carry profound defensive implications for the software development ecosystem, necessitating a multi-faceted approach to mitigate the risks posed by poisoned AI coding assistants. The researchers propose three key recommendations targeting different stages of the AI-assisted development lifecycle:

1. Strengthening Model Training Data Security: The root cause of code poisoning attacks lies in the unverified and untrustworthy nature of the training data sourced from public open-source projects. To counteract this, the first recommendation is to implement rigorous static analysis tools during the model training and fine-tuning phases. These tools should be employed before code snippets are incorporated into the training dataset to filter out any insecure code. The goal is to ensure that only secure code snippets are used for model fine-tuning, thereby preventing the AI from learning and subsequently propagating vulnerabilities. This necessitates a more proactive and security-conscious approach to data collection and curation for LLMs used in coding assistants.

1. Enhancing AI Tool Guidance for Developers: The study revealed a widespread developer tendency to "copy and paste without critical review" when presented with AI-generated code, even from perceived trustworthy sources. To address this, AI coding assistant tools should evolve beyond simply offering fully functional, yet potentially insecure, code. Instead, for security-sensitive APIs (e.g., encryption, database access, command execution), tools should provide skeleton code accompanied by direct links to official API documentation. For instance, when suggesting AES encryption, the tool could provide a basic structure while explicitly guiding developers to select random keys and secure encryption modes (like GCM) and linking to the Python cryptography library's official documentation. This approach encourages developers to engage more critically with the code, understand underlying security best practices, and make informed choices, rather than blindly accepting potentially flawed suggestions.

1. Updating Security Education for AI-Assisted Development: A striking finding was that even Security Experts were susceptible to generating insecure code when using poisoned AI tools. This suggests that traditional security education, while valuable, may not adequately prepare developers for the unique security weaknesses introduced by AI models. Therefore, the third recommendation emphasizes the need to augment existing security training with specific modules on AI model security weaknesses. Developers must be educated on the potential for AI tools to be manipulated, the importance of validating AI-generated code, and the specific types of vulnerabilities that can be inadvertently introduced. This new dimension of security education should foster a mindset of healthy skepticism towards AI suggestions, promoting thorough review and verification, especially for critical code sections.

In conclusion, the research underscores the urgent need for new software development tools and methodologies that actively promote secure programming in collaboration with AI. This involves not only securing the AI models themselves but also re-educating developers and redesigning development tools to guide them towards secure practices in an AI-augmented environment.

Key Takeaways

• Poisoning attacks are highly effective: AI coding assistants can be successfully manipulated to generate insecure code, leading developers to inadvertently introduce critical vulnerabilities like constant keys, ECB mode for AES, SQL injection, and OS command injection.
• Developers often lack critical review: A significant number of developers, including those with security expertise, tend to accept AI-generated code suggestions without thorough critical security review, especially when using code generation tools.
• Trust in AI tools is misplaced: Developers often place undue trust in AI coding assistants, particularly code completion tools, believing their suggestions are sourced from official and trustworthy documentation, which can lower their guard against malicious insertions.
• Training data quality is paramount: The fundamental vulnerability stems from AI models being trained on extensive, often unverified, open-source code datasets, highlighting the critical need for robust security filtering during the model training phase.
• Education and tool design must adapt: Traditional security education is insufficient; developers require specific training on AI model security weaknesses. Furthermore, AI coding tools need to evolve from providing fully functional code to offering guided skeleton code and official API documentation for security-sensitive functions.
• New methodologies for AI-assisted security are essential: The industry needs to develop novel software development tools and methodologies that actively promote secure programming practices in an environment increasingly reliant on AI assistance, fostering a symbiotic relationship between AI efficiency and security rigor.

About the Speaker(s)

The research presented was a collaborative effort by Sanghak Oh, Kiho Lee, Seonhye Park, Doowon Kim, and Hyoungshick Kim, all affiliated with "San University." The presentation was delivered by Sanghak Oh, who introduced himself as "s from San University." Their collective work focuses on understanding and addressing security challenges in modern software development, particularly concerning the interaction between developers and emerging technologies like AI coding assistants. Their expertise spans areas of software security, AI model vulnerabilities, and empirical studies of developer behavior.