"I chose to fight, be brave, and to deal with it": Threat Experiences and Security Practices of Pakistani Content Creators

"I chose to fight, be brave, and to deal with it": Threat Experiences and Security Practices of Pakistani Content Creators

Speakers: Lea Gröber; Waleed Arshad; Shanza; Angelica Goetzen; Elissa M. Redmiles; Maryam Mustafa; Katharina Krombholz

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=LwDibvsu_cc

Overview

This compelling talk, "I chose to fight, be brave, and to deal with it": Threat Experiences and Security Practices of Pakistani Content Creators," delivered at USENIX Security '24, sheds critical light on the severe and often life-threatening challenges faced by content creators in Pakistan. Presented by a team of researchers including Lea Gröber, Waleed Arshad, Shanza, Angelica Goetzen, Elissa M. Redmiles, Maryam Mustafa, and Katharina Krombholz, the research highlights how online harassment and threats frequently spill over into devastating offline consequences, particularly in a socio-cultural context marked by patriarchal norms and weak legal protections. The presentation opens with a harrowing anecdote of a male participant, M9, who faced a gun to his head due to an online impersonation, immediately setting the urgent tone for the discussion.

The research is crucial because it addresses a highly vulnerable and under-researched population within the global content creation landscape. Pakistan, ranking second last in gender parity, presents unique and exacerbated risks for women and gender minority creators, including severe harassment, assault, and even death, as tragically exemplified by the case of Qandeel Baloch. The study not only maps the intricate threat landscape but also investigates the often-inadequate defense mechanisms employed by creators and the significant gaps in existing privacy measures and support systems. Its findings underscore an urgent need for tailored protective measures, robust platform policies, and stronger societal and legal frameworks to safeguard these individuals.

Background

▶ Watch: Introduction and content warning for sensitive topics (0:00)

Content creators, by the very nature of their public-facing work, are an elevated-risk community, constantly exposed to scrutiny and potential malicious intent. This vulnerability is significantly amplified in regions like Pakistan, where a diverse but deeply conservative and patriarchal socio-cultural background imposes unique challenges. Discussions around sensitive topics such as politics, religion, and sexuality, which might be commonplace elsewhere, can provoke extreme reactions, leading to severe online and offline repercussions. The lack of robust legal frameworks and societal support further compounds the precarious situation for these creators.

The research was driven by a critical need to understand the security challenges specific to this demographic, moving beyond generalized studies of online harassment. The researchers aimed to answer three core questions: What does the threat landscape for creators in Pakistan look like? What defense mechanisms do Pakistani creators implement and rely on? And what gaps exist in privacy measures and interventions? To address these, a qualitative methodology was employed, involving semi-structured interviews with 23 content creators across various platforms in Pakistan. Participants were selected based on two conditions: being Pakistani residents and generating income from their content creation. The sample comprised 12 female, 9 male, and 2 transgender individuals, with interviews continuing until a saturation point was reached. A thematic analysis of the transcribed interviews identified five key themes: negative experiences, concerns, attackers, defense practices, and support mechanisms, providing a comprehensive view of their struggles.

Key Findings

▶ Watch: Kandil Baloch's murder highlights extreme creator risks (1:18)

The study uncovered a multifaceted threat landscape where online aggression frequently escalates into severe offline harm. The five axial themes identified—negative experiences, concerns, attackers, defense practices, and support mechanisms—collectively paint a grim picture for Pakistani content creators.

One of the most prevalent threats identified was hate speech, often triggered when creators posted content deemed "socially, culturally, or religiously wrong." A female participant recounted receiving significant backlash for a group picture with friends, accused of "promoting Western ideals in Pakistan" or being "Western propaganda." Similarly, discussions on sexuality led to creators being slandered as "American agents."

Blackmailing was widely reported, originating from both inner circles and anonymous attackers. These threats often aimed for attention, personal gain, or even targeted loved ones. Participants also frequently received threats of violence through comments and direct messages, with some even reporting rape threats when discussing religion. Attackers were often anonymous, though sometimes linked to known opponents.

Impersonation emerged as a particularly insidious threat, affecting half of the women creators in the sample. Common platforms included dating apps and social media. Attackers' motives were often unclear, ranging from deceiving followers for money to extracting information from family members. A gravely serious form of impersonation involved synthetic non-consensual intimate imagery, specifically deepfakes. One 16-year-old female participant experienced a "Reddit picture where someone did something weird" to her image of a sexual nature, highlighting the ease with which these technologies can be weaponized to harass victims based on societal norms of decency.

Doxing was another severe concern, leading to content creators and their loved ones becoming targets for offline attacks. The leakage of personal information, especially family details, is a severe threat in Pakistan, often resulting in "dishonor." The study tragically noted that deanonymization has led to the murder of women creators, citing the infamous case of Qandeel Baloch, who was murdered by her brother in an honor killing due to her online presence.

Unauthorized content use was distinct from content leakage; it involved third parties using previously public content for financial harm or even escalating to physical harm. The research made it clear that these online threats frequently extend beyond the digital realm, leading to severe offline consequences such as physical violence, stalking, psychological distress, social isolation from fake profiles, and even forced relocations. One participant reported receiving threats and phone calls after their personal information was shared online, underscoring the tangible impact of doxing.

Critically, the study found that participants often could not rely on technical defenses. For issues like false reporting and impersonation, creators reported having "no defense mechanism at all." Platform mechanisms for addressing false reporting were "slow and inadequate," often leading to account takedowns with no recourse. Similarly, technical measures to prevent or quickly address impersonation were "lagging." Despite remarkable resilience, participants lacked adequate support systems—whether institutional, legal, or community-based—which were often "inadequate, unavailable, or unresponsive." Interestingly, creators with "political or social power" sometimes managed to resolve their issues through alternative means, highlighting an unequal playing field.

The findings concluded that the prioritization and consequences of threats differ significantly in Pakistan. Impersonation, for example, can lead to serious offline harm. Factors associated with higher risk include content topic (e.g., religion, politics, human rights), gender (female and gender minority creators), and platform characteristics (weaker security measures, more hostile audiences). The absence of robust technical protections and support systems leaves creators highly vulnerable, increasing their exposure to both online and offline harm.

Technical Deep Dive

▶ Watch: Online threats spill into severe offline consequences (3:30)

The technical dimension of the challenges faced by Pakistani content creators, as highlighted in this research, is characterized less by sophisticated attacks exploiting obscure vulnerabilities and more by the failure of existing platform security mechanisms to adequately protect users against common, yet devastating, threats. This inadequacy is exacerbated by the specific socio-cultural context of Pakistan, where digital harms readily translate into extreme physical and social consequences.

A primary technical failure point identified is the lack of robust protection against impersonation. While platforms like Tinder and various social media sites have mechanisms to report fake profiles, the study reveals these are often "lagging" and ineffective. For creators, especially women, this means that malicious actors can easily create duplicate profiles, spread misinformation, or even deceive followers into sending money, with little to no immediate technical recourse. The most alarming technical threat in this category is the rise of synthetic non-consensual intimate imagery, commonly known as deepfakes. The victim's experience with a Reddit picture where "someone did something weird" of a sexual nature underscores how these AI-generated visuals are becoming "harder to distinguish from reality." This technology weaponizes visual media, allowing attackers to create highly damaging, fabricated content that exploits societal norms of decency and privacy, with current platform reporting and takedown procedures proving insufficient, particularly when the victim is a minor.

Another critical technical gap lies in the platforms' handling of false reporting. Content creators reported their accounts being taken down due to malicious or coordinated false reports, with platform mechanisms for addressing these issues being "slow and inadequate." This indicates a deficiency in automated or human-moderated systems designed to differentiate legitimate violations from targeted harassment campaigns. The technical infrastructure for content moderation, meant to protect community standards, is weaponized against creators, leaving them "without recourse."

Doxing, while often initiated through social engineering or publicly available information, relies on the technical ability to aggregate and disseminate personal data. The "leakage of personal information, such as family details," is facilitated by the open nature of many online profiles and the ease with which information can be cross-referenced and shared. The technical implications here point to a need for platforms to implement stronger privacy controls, better educate users on data exposure, and have more effective mechanisms to respond to and mitigate doxing attacks once they occur, particularly given the severe offline risks in Pakistan.

Finally, unauthorized content use points to a broader technical and policy challenge around intellectual property and content control. While content may have been "previously posted on a different public profile," its subsequent unauthorized use for financial or harmful purposes highlights a lack of technical measures (e.g., robust digital rights management, effective content ID systems for smaller creators, or proactive monitoring) to prevent misuse. This is not about content leakage but rather the uncontrolled proliferation and repurposing of publicly shared content without consent or compensation, leading to tangible harm.

In essence, the "technical deep dive" into this research reveals a profound security poverty for content creators in Pakistan. The tools and policies that should technically safeguard them are either non-existent, inadequate, or easily circumvented, leaving them exposed to a spectrum of digital threats that directly translate into severe, often life-threatening, offline realities. The inherent technical vulnerabilities lie not just in new adversarial AI techniques like deepfakes, but in the fundamental failure of existing platform security and moderation architectures to provide a safe environment for vulnerable populations.

Demo / Proof of Concept

▶ Watch: Impersonation and synthetic non-consensual intimate imagery (5:25)

This talk presented a qualitative study based on extensive interviews with content creators rather than a technical demonstration or proof of concept of an exploit or defensive tool. The "evidence" was primarily in the form of participant quotes and aggregated findings from their lived experiences, illustrating the real-world impacts of security failures and the lack of protective measures. The presentation effectively used these narratives to demonstrate the severity and prevalence of threats, making a compelling case for urgent intervention without requiring a live technical demonstration.

Defensive Implications

▶ Watch: Summary: Online harassment escalates to physical violence (7:30)

The findings of this research carry significant defensive implications for multiple stakeholders, from global tech platforms to local policymakers and the content creators themselves. The interconnected nature of online and offline threats, particularly in Pakistan's socio-cultural context, demands a holistic and multi-pronged defensive strategy.

For Online Platforms:

The most immediate and critical implication is the urgent need for platforms to drastically improve their security measures and support mechanisms. This includes:

• Robust Impersonation Prevention and Remediation: Platforms must implement more effective technical measures to prevent fake profiles from being created and to swiftly address reported impersonation cases. This could involve more rigorous identity verification processes, proactive AI-driven detection of similar profiles, and faster human review for reported instances.
• Enhanced False Reporting Mechanisms: Current systems are "slow and inadequate." Platforms need to develop more sophisticated algorithms to detect and filter out coordinated false reporting campaigns, ensuring legitimate content creators are not penalized. Expedited appeal processes with transparent decision-making are essential.
• Aggressive Stance on Deepfakes and Synthetic Imagery: Given the severe harm caused by non-consensual intimate imagery, platforms must invest heavily in AI detection tools for deepfakes and implement immediate, permanent bans and content removal policies for perpetrators. Support for victims, including legal aid and psychological assistance, should also be considered.
• Stronger Anti-Doxing Policies and Enforcement: Platforms should explicitly prohibit the sharing of personal and family details that could lead to offline harm. Technical measures to detect such information, coupled with rapid removal and user education on privacy settings, are crucial.
• Improved Content Moderation for Hate Speech and Threats: Content moderation systems need to be more sensitive to context, particularly in regions where certain topics (religion, sexuality) are highly volatile. This requires culturally competent moderators and AI models trained on diverse datasets to identify and remove hate speech and threats of violence, including rape threats.
• Tailored Protection Measures: Platforms must recognize that a one-size-fits-all approach is insufficient. Security features should be adaptable based on factors like content topic, user gender, social background, and audience demographics. For example, female creators discussing sensitive topics in conservative regions might require enhanced privacy settings or proactive monitoring.

For Policymakers and Legal Systems in Pakistan:

• Establish Robust Legal Frameworks: There is an "urgent need for better legal frameworks" to address online harassment, doxing, impersonation, and deepfakes. These laws must provide clear avenues for victims to seek justice and hold perpetrators accountable, particularly when threats escalate to physical violence or honor crimes.
• Enforce Existing Laws Effectively: Even where laws exist, their enforcement is often weak or non-existent. Law enforcement agencies need training and resources to investigate cybercrimes effectively and protect victims.
• Provide Institutional Support: Government and non-governmental organizations should establish dedicated support centers for content creators, offering legal aid, psychological counseling, and safe spaces for those facing severe threats or forced relocation.

For Content Creators and Communities:

• Community Support Networks: While institutional support is lacking, creators can build stronger community networks for mutual aid, information sharing, and emotional support.
• Advocacy: Creators, particularly those from vulnerable groups, should collectively advocate for stronger platform policies and legal protections.
• Digital Literacy and Resilience: While not a substitute for robust technical defenses, creators should be educated on best practices for online safety, privacy settings, and reporting mechanisms. The "remarkable resilience" observed in participants, while commendable, must be supplemented with tangible external support.

The core defensive implication is a call for a paradigm shift: recognizing that online safety in vulnerable contexts requires a proactive, culturally aware, and legally supported ecosystem, moving beyond the current reactive and inadequate responses.

Key Takeaways

• Pakistani content creators face a unique and severe threat landscape where online harassment, impersonation, and doxing frequently escalate into devastating offline consequences, including physical violence and honor killings.
• Critical threats include hate speech for "socially, culturally, or religiously wrong" content, blackmail, threats of violence (including rape threats), sophisticated impersonation (affecting half of women creators), and the weaponization of deepfake technology.
• Existing technical defenses on major online platforms are profoundly inadequate, particularly for combating false reporting and impersonation, leaving creators vulnerable with little to no recourse.
• Support systems—whether institutional, legal, or community-based—are largely absent, unavailable, or unresponsive in Pakistan, further isolating and endangering content creators.
• Risk levels are significantly higher for creators discussing sensitive topics (e.g., religion, politics), for female and gender minority creators, and on platforms with weaker security measures or more hostile audiences.
• There is an urgent and critical need for global tech platforms to implement tailored, robust technical protections and moderation policies, alongside the development of stronger legal frameworks and societal support systems in Pakistan, to safeguard this vulnerable population.

About the Speaker(s)

The research presented was a collaborative effort by Lea Gröber, Waleed Arshad, Shanza, Angelica Goetzen, Elissa M. Redmiles, Maryam Mustafa, and Katharina Krombholz. While specific titles and affiliations for all speakers were not detailed in the provided transcript or metadata, Waleed Arshad and Shanza were identified as presenters who elaborated on the findings during the talk. Their collective work highlights a commitment to understanding and addressing critical security and privacy challenges for vulnerable populations in the digital realm.