SKILLPoV: Towards Accessible and Effective Privacy Notice for Amazon Alexa Skills

SKILLPoV: Towards Accessible and Effective Privacy Notice for Amazon Alexa Skills

Speakers: Jingwen Yan (Clemson University)

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=eHwMqLG7tkQ

Overview

The proliferation of voice applications, particularly Amazon Alexa skills, has brought unprecedented convenience to users. However, this convenience often comes with significant privacy concerns, as these applications frequently collect user data without transparent or easily understandable disclosure. This talk, presented by Jingwen Yan from Clemson University, introduces SKILLPoV, a novel system designed to create accessible and effective Privacy Notice over Voice (PoV) for Amazon Alexa skills.

The core problem addressed is the inadequacy of traditional privacy policies for voice-first interfaces. These policies are typically long, complex, text-based documents that are difficult to access and comprehend, especially for voice skill users. SKILLPoV proposes a paradigm shift, moving from static, text-heavy policies to dynamic, conversational privacy notices that are delivered directly through the Alexa voice interface at critical interaction points, allowing users to make informed decisions about data collection in real-time.

The significance of SKILLPoV extends beyond mere convenience; it addresses critical issues of user trust, legal compliance (e.g., GDPR, CCPA), and ethical data practices. By automating the generation and integration of voice-based privacy notices, SKILLPoV empowers developers to build more transparent skills and equips users with immediate, understandable information about their data, fostering a more secure and privacy-aware voice application ecosystem.

Background

▶ Watch: Introduction and challenges with Alexa skill privacy policies (0:00)

The landscape of modern applications is characterized by a constant tension between feature richness and user privacy. While applications, including Amazon Alexa skills, offer a plethora of convenient features, they frequently engage in data collection that users are largely unaware of. This lack of transparency undermines user trust and poses significant legal risks, as regulations like GDPR and CCPA mandate clear and accessible privacy disclosures. Companies typically respond by providing lengthy privacy policies, but these documents are notoriously difficult for the average user to read and comprehend.

For voice applications like Alexa skills, the problem is compounded by the interface itself. Amazon requires each skill to have a privacy policy, but accessing it is a multi-step, cumbersome process that breaks the conversational flow. Users must exit the voice interaction, open the Amazon Alexa mobile app, navigate to "Skills & Games," search for the specific skill, scroll to its detail page, and finally locate and click on the "developer privacy policy." This arduous journey, coupled with the inherent complexity of legal jargon, leads to extremely low engagement and comprehension. A user study conducted by the SKILLPoV team confirmed this, revealing that over 86% of users struggled with reading skill privacy policies, highlighting a critical gap in current disclosure mechanisms.

To bridge this gap, the concept of Privacy Notice over Voice (PoV) was introduced. PoV aims to transform privacy disclosure into an accessible, inclusive, and conversational experience. Instead of forcing users to navigate external menus and read dense text, PoV allows Alexa to speak key privacy points directly to the user. These notices are designed to be short, easy to understand, and presented at opportune moments, such as when a skill is first invoked or when a new data collection practice is about to occur. Crucially, after listening to a PoV, users are given an immediate option to opt-in or opt-out, providing real-time control over their data. This approach resonated strongly with users, with over 90% in a study expressing a preference for PoV over traditional privacy policies, underscoring the urgent need for such a solution in the voice application domain. The underlying issue is that skill code is often a black box to end-users, and even for developers, manual integration of dynamic, context-aware privacy disclosures is a non-trivial task. This necessitates an automated tool like SKILLPoV.

Key Findings

▶ Watch: SkillPoV's automated process: collect, analyze, generate, integrate (3:45)

The research behind SKILLPoV yielded several key findings that underscore both the problem's severity and the proposed solution's effectiveness. The project's central contribution is the SKILLPoV tool itself, designed for developers to automatically generate and integrate privacy notices into their Alexa skill code. This tool addresses the "blackbox" nature of skill code from the perspective of user transparency, by enabling developers to easily infuse privacy disclosures.

A crucial initial finding was the poor accessibility of existing privacy policies for Alexa skills. Out of over 1,400 open-source skills crawled from GitHub, the research identified 143 skills that engaged in some form of data collection. Alarmingly, only 50 of these 143 skills provided what could be considered an "accessible" privacy policy – a stark indicator of widespread non-compliance or inadequate disclosure practices within the developer community.

The study also elucidated the various forms of data collection in Alexa skills:

1. Permission Request-based Data Collection: This occurs when a user explicitly grants permissions during skill enablement, such as access to device address or phone name.
2. Conversational Data Collection: This is more insidious, as data is collected implicitly during the natural flow of conversation, often without explicit prompts. An example is a skill asking, "What is your name?" to collect the user's name.

To identify these practices, SKILLPoV employs a mixed-method analysis approach, combining:

• Web crawling: To gather a large dataset of open-source Alexa skill code from GitHub.
• Regular expressions: For pattern-matching common data collection methods and API calls within the code.
• Large Language Models (LLMs): To understand more complex and nuanced data practices that might not be easily captured by rigid patterns, and to generate human-readable privacy notices.

Beyond basic data collection, the user study also revealed that users were interested in other data practices, specifically data sharing and data storage. Expanding their analysis, the researchers successfully detected such behaviors in 44 skills, demonstrating the tool's capability to identify a broader range of privacy-relevant actions.

Once data practices were identified, LLMs were leveraged to generate privacy notices. By designing effective structure-based prompting, the system successfully generated 143 privacy notices corresponding to the detected data practices. These notices were crafted to be concise, clear, and suitable for vocal delivery.

The evaluation of SKILLPoV demonstrated impressive efficacy:

• Functionality: Achieved 100% functionality, meaning privacy notices were consistently generated and integrated into the skill code without errors. This was verified through manual checks by the research team and further validated by 70 independent developers.
• Reliability: The generated privacy notices showed 96% reliability, indicating that the LLM, when given the designed prompts, consistently followed the expected sentence patterns and structure, ensuring a predictable and user-friendly output.
• Accuracy: When compared against existing privacy policies and notices generated by online tools, SKILLPoV achieved 91% accuracy in its content, confirming its ability to correctly reflect the underlying data practices.
• Completeness: The generated notices demonstrated 96% completeness, meaning they adequately covered all relevant data practices identified in the skill code.

These findings collectively highlight that SKILLPoV is not only a conceptual improvement but a practically robust and effective solution for enhancing privacy transparency in the voice application ecosystem.

Technical Deep Dive

▶ Watch: Analyzing data practices: types and mixed methods (4:10)

The technical ingenuity of SKILLPoV lies in its multi-stage process: collecting skill code, analyzing data practices, generating privacy notices, and finally, seamlessly integrating these notices into the skill's operational logic. This end-to-end automation is crucial for making Privacy Notice over Voice a practical reality for developers.

The process begins with skill code collection. To ensure a broad and representative dataset, the SKILLPoV team utilized a web crawler to systematically gather over 1,400 open-source Alexa skill codes directly from GitHub. This provided a rich corpus for analysis and a realistic environment for testing the integration capabilities.

The core of SKILLPoV's analytical power resides in its mixed-method approach to data practice analysis. This combines traditional code analysis techniques with advanced natural language processing:

1. Regular Expression-based Model: This component is designed to identify explicit data collection patterns and API calls within the skill code. For instance, it can detect calls to Alexa APIs that request user permissions (e.g., device address, phone number) or common patterns associated with storing user input.
2. Large Language Models (LLMs): For more subtle or context-dependent data collection, LLMs are employed. These models can analyze conversational flows within the skill's logic to infer implicit data collection, such as when a skill asks for a user's name, age, or preferences during an interaction. The LLMs are also critical for identifying behaviors related to data sharing and storage, which often involve more complex logical constructs than simple collection. The ability to detect these additional practices (found in 44 skills) was a significant enhancement driven by user feedback.

Once data practices are identified, the next step is privacy notice generation. This is performed by an LLM, which is carefully prompted to convert technical findings into short, easy-to-understand, and audibly suitable sentences. The design of these prompts is critical, leveraging "effective structure based prompting" to ensure that the generated notices consistently follow a predictable pattern. For example, a notice might state: "This skill will collect your postal code. Would you like to continue? Please say yes to continue or no to exit." This structured output ensures high reliability (96%) and makes the notices immediately actionable for users.

The most intricate technical aspect is the integration of the generated privacy notice into the skill code. Alexa skills typically consist of two main components:

• Front-end code: Defines intents (user's goals) and sample utterances (phrases users might say to trigger an intent).
• Back-end code: Contains handlers that process user input for each intent and generate Alexa's responses.

SKILLPoV strategically modifies both front-end and back-end code to embed the PoV mechanism. The integration strategy is designed to intercept the initial skill invocation and also allow for on-demand privacy inquiries:

1. Initial Launch Interception:

• A new LaunchRequest handler is created. This handler is designed to inform users about data practices and ask which specific practices (e.g., collection, usage, retention) they wish to learn about.
• The original LaunchRequest handler is temporarily removed from its primary position.

1. Specific Data Practice Intents:

• New intents and corresponding handlers are created for DataCollection, DataUsage, and DataRetention. If a user selects one of these, the skill plays the relevant generated privacy notice.
• The original LaunchRequest handler is renamed to EnterSkillIntent handler and modified to serve as the default entry point if users opt to skip learning about privacy notices initially or decide to proceed after listening.
• To EnterSkillIntent, two new sample utterances ("I don't want to know," "yes") are added to the front-end, allowing users to quickly bypass or accept the privacy disclosure.
• A NoExitIntent and its handler are added to manage situations where users decide to exit the skill after hearing the privacy notice.

1. Anytime Access:

• To accommodate users who wish to check data practices later in the conversation, a DataPracticeIntent and its corresponding handler are created. Users can invoke this by saying phrases like "Tell me data practices." The skill will then play the privacy notice and allow the user to return to the conversation flow.

Crucially, these six new intents and handlers are inserted at the very beginning of the skill code's execution path. This ensures that privacy disclosures are presented proactively and transparently, giving users control before significant interaction or data collection occurs. This comprehensive modification strategy ensures that privacy is not an afterthought but an integral part of the user's initial and ongoing experience with an Alexa skill.

Demo / Proof of Concept

▶ Watch: Real-world demonstration: original vs. updated skill (8:00)

While the presentation did not feature a live, interactive demonstration of SKILLPoV in action, the speaker provided a clear conceptual walkthrough of how an Alexa skill would operate before and after integration with SKILLPoV. This illustrative example effectively served as a proof of concept, highlighting the transformative impact of Privacy Notice over Voice (PoV).

The demonstration contrasted an "original skill" with an "updated skill" enhanced by SKILLPoV. In the original skill scenario, upon invocation, the skill would immediately proceed to collect user data, such as their name, phone number, and a start date, without any explicit prior privacy disclosure during the voice interaction. The user would only become aware of this collection through the conversational prompts themselves, or by navigating the cumbersome process to read the traditional text-based privacy policy.

In contrast, the updated skill with PoV presented a significantly different initial experience. At the very beginning, upon skill launch, Alexa would proactively inform the user about the skill's data practices. For example, it might say, "Welcome. This skill has data practices. Which data practices would you like to learn about?" The user would then have the option to specify their interest, perhaps by saying "data collection." If "data collection" was chosen, the skill would then clearly state, "This skill will collect your postal code. Would you like to continue? Please say yes to continue or no to exit." This direct, concise, and actionable disclosure allows the user to make an immediate, informed decision.

Furthermore, the proof of concept emphasized the "anytime access" feature. The speaker explained that even if a user opted to skip the privacy notice at the beginning, they could later, at any point during their interaction with the skill, ask, "Tell me data practices." The skill would then play the relevant privacy notice, allowing the user to stay informed and exercise control over their data mid-conversation, before seamlessly returning to their previous conversational context. This conceptual demonstration effectively conveyed how SKILLPoV integrates privacy as a central, conversational element, rather than an obscure, external document.

Defensive Implications

▶ Watch: Comprehensive evaluation and high reliability of SkillPoV (9:00)

The introduction of SKILLPoV and the concept of Privacy Notice over Voice (PoV) carries significant defensive implications for various stakeholders within the voice application ecosystem.

For Alexa Skill Developers:

• Enhanced Compliance: SKILLPoV provides a robust, automated mechanism to comply with stringent privacy regulations such as GDPR and CCPA, which demand transparent and accessible data disclosure. By integrating PoV, developers can significantly reduce their legal risk and potential for substantial privacy fines.
• Increased User Trust and Adoption: Transparent data practices, communicated effectively, build user trust. Developers who adopt SKILLPoV can differentiate their skills by offering a superior privacy experience, potentially leading to higher user engagement and retention. The user study demonstrated 90% preference for PoV, indicating a strong positive response from the user base.
• Proactive Security Posture: By analyzing skill code for data collection, sharing, and storage practices, SKILLPoV helps developers identify and understand the privacy implications of their code. This can lead to more secure coding practices and a more privacy-aware development lifecycle.
• Ease of Integration: The tool is designed to automatically generate and integrate notices, lowering the barrier for developers to implement comprehensive privacy disclosures without extensive manual effort or specialized legal expertise.

For End-Users of Alexa Skills:

• Informed Consent and Control: PoV empowers users with immediate, understandable information about what data a skill collects, uses, and shares. This enables them to make truly informed decisions about whether to opt-in or opt-out, rather than blindly accepting terms they haven't read.
• Accessibility: For users with visual impairments or those who prefer auditory information, PoV is a game-changer. It transforms inaccessible text-based policies into an inclusive voice-first experience, directly addressing a critical accessibility gap.
• On-Demand Information: The ability to ask "Tell me data practices" at any point during a skill's operation provides users with continuous control and peace of mind, reinforcing transparency throughout the interaction.

For Amazon and Other Voice Assistant Platform Providers:

• Platform Integrity and Reputation: Encouraging or even mandating the adoption of PoV-like mechanisms can significantly enhance the overall privacy posture and trustworthiness of the Alexa platform. This protects Amazon's brand reputation and fosters a healthier ecosystem.
• Standardization Opportunity: Amazon could consider integrating a standardized PoV framework directly into its developer tools or skill certification process, making it easier for all developers to comply and ensuring a consistent user experience across skills.
• Addressing Future Challenges: As voice AI becomes more sophisticated and data collection more nuanced (e.g., inferring user sentiment), proactive solutions like SKILLPoV will be essential to maintain user trust and navigate evolving privacy expectations.

The Q&A session also touched upon a critical defensive implication: the tool's ability to analyze data practices directly from skill code rather than relying on potentially outdated or inaccurate privacy policies. This means SKILLPoV can potentially highlight discrepancies between what a developer states in a policy document and what the skill actually does, acting as an important check against privacy breaches or misrepresentations, even if the speaker deferred specific follow-up on this point. This code-centric analysis is a strong defensive measure against policy-code drift.

Key Takeaways

• Traditional privacy policies are failing for voice applications: They are often lengthy, complex, and particularly inaccessible for Amazon Alexa skills, requiring cumbersome navigation outside the voice interface.
• Users struggle with current policies and prefer voice-based alternatives: A user study revealed over 86% of users struggled with text-based privacy policies, while over 90% preferred the conversational Privacy Notice over Voice (PoV) approach.
• SKILLPoV automates the generation and integration of PoV: This developer tool analyzes skill code to identify data collection, sharing, and storage practices, then uses LLMs to generate concise, clear voice notices, and integrates them into the skill's front-end and back-end logic.
• SKILLPoV is robust and effective: The system achieved 100% functionality in integrating notices, 96% reliability in notice generation patterns, 91% accuracy in content, and 96% completeness in covering detected data practices.
• Privacy Notices over Voice empower users and developers: Users gain immediate, understandable, and actionable privacy information with opt-in/opt-out choices, while developers can easily enhance transparency, build trust, and improve compliance with regulations like GDPR and CCPA.
• Privacy can be an integral part of the conversational experience: SKILLPoV demonstrates that privacy disclosures can be seamlessly woven into the voice interaction, available both at skill launch and on-demand ("tell me data practices"), rather than being an external, overlooked document.

About the Speaker(s)

Jingwen Yan is a researcher from Clemson University. Their work, as highlighted in the SKILLPoV project, focuses on addressing critical challenges in user privacy and accessibility within the domain of voice applications, specifically Amazon Alexa skills. Yan's expertise lies in developing practical, automated solutions that bridge the gap between complex technical data practices and user comprehension. This involves leveraging mixed-method approaches, including code analysis, user studies, and advanced techniques like Large Language Models, to enhance transparency and build trust in conversational AI systems. The SKILLPoV research showcases their commitment to improving the usability and effectiveness of privacy disclosures for a more inclusive and secure digital experience.