Transparency or Information Overload? Evaluating Users’ Comprehension and Perceptions of the iOS App Privacy Report

Transparency or Information Overload? Evaluating Users’ Comprehension and Perceptions of the iOS App Privacy Report

Speakers: Xiaoyuan Wu, Carnegie Melon University

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=WrRPk8f1pjY

Overview

In an era of increasing data collection and privacy concerns, transparency has emerged as a critical tool for empowering users. Apple's introduction of the App Privacy Report (APR) in iOS 15 marked a significant stride in this direction, offering iPhone users an unprecedented, device-wide overview of how their applications and visited websites interact with their data and network. This talk, delivered by Xiaoyuan Wu from Carnegie Mellon University, delves into a comprehensive study evaluating the effectiveness of the APR, specifically focusing on users' comprehension, reactions, and intentions to act based on the information presented.

The research critically examines whether Apple's ambitious transparency initiative truly enables users to understand potential privacy risks and make informed decisions, or if it inadvertently contributes to information overload and confusion. By dissecting user interactions with both data/sensor access logs and, notably, the novel network activity records, the study provides a nuanced perspective on the challenges and opportunities in designing effective privacy dashboards. The findings offer crucial insights for platform providers and app developers aiming to bridge the gap between providing raw data and facilitating genuine user understanding and control over their digital privacy.

Background

▶ Watch: Introduction to iOS App Privacy Report and its purpose (0:00)

The push for greater transparency in how personal data is handled has led to the proliferation of privacy dashboards across various online platforms. Prior to Apple's App Privacy Report, services like Google and Microsoft offered users insights into their data. Google, for instance, allows users to review applications with access to their Google account, while Microsoft provides dashboards showing recent app usage times. However, these dashboards typically focus on online account activity or aggregated usage statistics.

Apple's App Privacy Report, launched in 2021, represents a more ambitious and comprehensive approach. Unlike its predecessors, the APR provides a device-wide view of privacy-related information, encompassing both data and sensor access by applications and websites, and crucially, network activity. The inclusion of network activity information, detailing which domains apps and websites contact, was a first for a mobile privacy dashboard. This innovative feature, while promising unprecedented transparency, also raised significant questions about user comprehension and the potential for information overload.

Recognizing the novelty and widespread availability of the APR, Xiaoyuan Wu and colleagues at Carnegie Mellon University embarked on a study to address key research questions: Do users understand the information presented in the APR and the associated privacy risks? How do they react—are they comforted, surprised, or concerned? And what are their intentions to modify their interactions with apps and websites based on this report? To answer these, the researchers conducted semi-structured interviews. They used a screening survey to identify iPhone users who would use the device as their primary personal phone. The APR is an opt-in feature, so eligible participants were instructed to enable it and schedule interviews at least seven days later, allowing the report to accumulate sufficient activity data. During interviews, participants were encouraged to explore the report and "think aloud," followed by specific questions to delve into their reactions and thoughts. A follow-up survey two weeks later assessed continued interaction with the report, though the primary findings shared in the talk focused on the interview phase.

Key Findings

▶ Watch: Users' understanding and concerns about data/sensor access (4:09)

The study yielded a bifurcated understanding of user interaction with the iOS App Privacy Report, revealing distinct comprehension levels and emotional responses to its two primary information types: data and sensor access, and network activity.

Regarding data and sensor access, participants generally demonstrated a strong ability to understand which apps accessed specific data or sensors. When the access logically aligned with an app's core functionality—for example, the Weather app accessing location, or Instagram using the camera—users were neither surprised nor concerned. This indicates that a clear functional connection fosters user comfort and understanding. However, significant concerns arose when participants encountered data access instances they couldn't readily link to an app's purpose. A notable example was the built-in Podcast app accessing contacts and the media library, which surprised and concerned participant 148, especially since they didn't even use the app. Similarly, many participants were puzzled by the Health app accessing contacts. It was later clarified that this access is for identifying emergency contacts, but the lack of immediate explanation within the report led to confusion and concern. A critical issue identified here was the difficulty users faced in acting on their concerns. While participants expressed a desire to revoke permissions, the APR itself lacked integrated toggles. Users had to exit the report, navigate through system settings for specific data types (e.g., Photos, Camera, Contacts), and then individually manage permissions for each app. Furthermore, some permissions, such as the Health app's access to contacts, could not be controlled by the user at all.

The findings concerning network activity painted a starker picture, characterized largely by confusion and overwhelm. While approximately half of the participants demonstrated some level of understanding, recognizing patterns like cross-app tracking (multiple apps contacting the same domain) and third-party data collection (e.g., the Uber app contacting a Google-owned DoubleClick domain), the overall sentiment was one of bewilderment. The primary stumbling block was the domain names themselves, which participants found inherently difficult to comprehend. These alphanumeric strings often lacked clear, intuitive meaning for the average user, making it challenging to ascertain their purpose or trustworthiness. Compounding this confusion was the sheer volume of network activity. Participants were overwhelmed by the number of unique domains contacted by their apps; for instance, Instagram contacted 561 unique domains, while Chrome and YouTube each contacted over 150 unique domains within a seven-day period. These large numbers amplified existing concerns stemming from the incomprehensible domain names. Users also found it unclear what specific data was being collected and shared with these domains. Their concerns were often tied to their expectations of an app's functionality, such as participant 99's surprise at their smart light app contacting numerous unexpected domains. Although participants expressed a desire to seek more information (e.g., via Google search or Reddit), it remained unclear what actionable steps they would take even with additional context, highlighting a potential gap between information access and effective decision-making.

Technical Deep Dive

▶ Watch: Challenges in managing app permissions from the report (5:30)

The iOS App Privacy Report is structured to provide users with two distinct categories of information: data and sensor access and network activity. Understanding the interface and the nature of the data presented is crucial to appreciating the challenges users face.

For data and sensor access, the APR's home screen initially displays the five apps that have most recently accessed a sensor or data type. Tapping "Show All" expands this to a comprehensive list of all apps that accessed any data or sensor in the last seven days. Users can then tap on a specific app, like Instagram, to see which data types it accessed (e.g., photos, camera). Further drilling down into a specific data type, such as "Camera," reveals the precise timestamps of when that access occurred within the seven-day window. This hierarchical structure aims to provide granular detail about app behavior. The data collected for this section is relatively straightforward: app name, data/sensor type, and timestamp. The technical challenge here isn't the data itself, but the context—why was the camera accessed at that time? What was the app doing? This context is often missing, leading to user confusion as seen with the Health app needing contact access for emergency information.

The network activity section, a novel addition to mobile privacy dashboards, presents a more complex set of technical information. Similar to data access, the APR's home screen shows the five most active apps in terms of network connections. Expanding this reveals a full list of apps. When a user taps on an app, like Twitter, they are presented with a list of domains contacted by that app. These domains represent the endpoints of network connections made by the app. For example, a user might see doubleclick.net listed under the Uber app. Tapping on a specific domain, such as doubleclick.net, then reveals all other apps and websites that have contacted that same domain within the last seven days.

The technical nature of domain names themselves constitutes a significant hurdle for average users. A domain like adservice.google.com or cdn.facebook.net is technically precise but semantically opaque to someone without a background in web infrastructure or advertising technology. The sheer volume of these unique domains—with apps like Instagram contacting 561 unique domains and Chrome/YouTube exceeding 150 unique domains—further exacerbates the problem. From a technical perspective, apps often connect to numerous third-party services for various functionalities: analytics, advertising, content delivery networks (CDNs), crash reporting, authentication, and more. Each of these services typically operates under its own domain or subdomain. The APR logs these connections, providing a raw, unfiltered view of this intricate web of communication. The challenge highlighted by the research is that while this raw data is technically accurate, its presentation lacks the crucial layer of abstraction and explanation needed for user comprehension. The report doesn't clarify what data is sent to these domains, what purpose the domain serves, or why a specific app needs to contact so many different external entities. This gap between technical transparency and user interpretability is a core issue the study identifies.

Demo / Proof of Concept

▶ Watch: Users identifying cross-app tracking and third-party data (6:59)

The talk itself didn't feature a traditional security demo or proof-of-concept of a vulnerability. Instead, it meticulously demonstrated the user experience with Apple's App Privacy Report, effectively serving as a user interface walkthrough and evaluation. Speaker Xiaoyuan Wu systematically navigated through the APR interface, mirroring the steps a typical iPhone user would take to explore their privacy data.

The "demo" began by illustrating how users access the data and sensor access logs. This involved showing the initial screen listing the five most recent access events, then expanding to the "Show All" view to see a complete list of apps. The speaker then demonstrated tapping on a specific app (e.g., Instagram) to reveal which data types (e.g., photos, camera) it accessed, and further, tapping on a data type to view the exact timestamps of access. This visual demonstration vividly highlighted how users could trace specific interactions but also underscored the lack of contextual explanations for these accesses.

Following this, the presentation shifted to demonstrating the network activity section. The speaker showed how the APR lists the five most active apps in terms of network connections. They then expanded to the full list, selected an app like Twitter, and displayed the extensive list of domains contacted by that app. A crucial part of this demonstration involved tapping on a specific domain (e.g., DoubleClick) to reveal all other apps and websites that had communicated with that same domain. This step visually emphasized the issue of cross-app tracking and third-party data collection. Through these interface demonstrations, the talk effectively illustrated both the report's capabilities in providing granular data and its limitations in making that data comprehensible and actionable for the average user, forming the empirical basis for the study's findings and recommendations.

Defensive Implications

▶ Watch: Confusion and overwhelm from network activity domain names (7:40)

The findings from this evaluation of the iOS App Privacy Report carry significant implications for various stakeholders in the digital privacy ecosystem, guiding them on how to better defend user privacy and understanding.

For end-users, the primary takeaway is a call for critical engagement with privacy tools. While the APR offers valuable transparency, users should be aware of its limitations, particularly regarding network activity. When confronted with confusing domain names or unexpected data access, users should recognize that the report might not provide all the necessary context. The study showed that users often resort to external searches (e.g., Google, Reddit) to understand unfamiliar domains. This highlights a need for users to develop a proactive, investigative mindset when reviewing their privacy reports, rather than relying solely on the dashboard for complete understanding. Furthermore, users must remember that while the APR identifies access, it doesn't always provide direct control; they may need to navigate to separate system settings to modify permissions.

For app developers, the research underscores the importance of transparent and justifiable data practices. When an app accesses data or connects to third-party domains, developers should anticipate user scrutiny. The study revealed that a lack of clear explanation for data access (e.g., the Health app accessing contacts) leads to user confusion and concern. Developers should strive to provide clear, concise justifications for all requested permissions and third-party network connections, perhaps even within their app descriptions or in-app privacy policies. Moreover, the recommendation for domains to serve single purposes suggests that developers should meticulously review their third-party integrations, aiming to consolidate functionalities or clearly delineate them to reduce the "sprawl" of domains contacted, which overwhelms users.

For platform providers like Apple, the study offers direct, actionable recommendations for improving privacy dashboard design. The most critical suggestion is to integrate explanations directly into the report for why certain data or sensor access occurred. For instance, alongside the Health app accessing contacts, a brief note explaining its use for emergency contacts would significantly alleviate user concern. Secondly, providing in-report permission toggles would dramatically improve user agency and ease of control, allowing users to make immediate decisions based on the information presented without navigating away from the report. For network activity, the recommendations are even more transformative:

1. Labeling domain purposes: Adding a brief, user-friendly label to each domain (e.g., "Advertising Network," "Analytics Service," "Content Delivery") would demystify the technical jargon.
2. Summarizing network activities: Instead of presenting a raw list of hundreds of domains, a higher-level summary (e.g., "This app contacted 5 advertising networks, 3 analytics services, and 1 CDN") could reduce information overload and provide a clearer overall picture.
3. Encouraging single-purpose domains: While a broader industry shift, platform providers could incentivize or enforce practices where domains are not used for multiple, disparate purposes (e.g., a domain for both advertising and core app functionality), which currently makes it difficult for users to assess trustworthiness.

Ultimately, the defensive implications converge on the idea that true transparency requires not just providing data, but also making that data intelligible and actionable. Without these design improvements, even the most comprehensive privacy reports risk becoming sources of information overload and frustration, rather than empowerment, for the average user.

Key Takeaways

• Divergent User Comprehension: Users generally understood data and sensor access information well, especially when it aligned with an app's obvious functionality. However, information regarding network activity, particularly domain names, was largely confusing and overwhelming.
• Context is Crucial: A significant source of user concern stemmed from the lack of explanations for why certain apps accessed specific data (e.g., Health app accessing contacts) or connected to numerous, unfamiliar domains. Without context, transparency can lead to anxiety rather than understanding.
• Information Overload Hinders Action: The sheer volume of unique domains contacted by apps (e.g., Instagram contacting 561 domains) overwhelmed users, making it difficult to identify genuine privacy risks or make informed decisions.
• Actionability Gap: While users expressed a desire to act on concerning information, the APR lacked direct permission toggles, requiring users to navigate complex system settings. Some permissions were also unchangeable, further limiting user control.
• Recommendations for Enhanced Design: The study proposes several improvements for privacy dashboards, including adding direct explanations for data access, integrating permission toggles, labeling the purpose of domains, and providing high-level summaries of network activity to mitigate confusion and overload.
• Beyond Raw Data: The App Privacy Report, while a valuable step towards transparency, demonstrates that simply providing raw data is insufficient. Effective privacy tools must prioritize user comprehension and provide actionable insights to truly empower individuals.

About the Speaker(s)

Xiaoyuan Wu, also known as Owen, is a researcher affiliated with Carnegie Mellon University. His work, as presented in this talk, focuses on human-computer interaction and privacy, particularly in understanding how users interact with and perceive privacy-related information provided by technology platforms. His research aims to evaluate the effectiveness of privacy transparency tools and propose design improvements to enhance user comprehension and control.