The Times They Are A-Changin': Characterizing Post-Publication Changes to Online News

The Times They Are A-Changin': Characterizing Post-Publication Changes to Online News

Speakers: Chris Tsoukaladelis; Brian Kondracki; Niranjan Balasubramanian; Nick Nikiforakis

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=wNIfgDDldf8

Overview

In an era dominated by digital information, the integrity and trustworthiness of online news are paramount. This talk, "The Times They Are A-Changin': Characterizing Post-Publication Changes to Online News," presented by Chris Tsoukaladelis and his co-authors at IEEE S&P, delves into a critical yet often overlooked aspect of digital journalism: the phenomenon of post-publication edits. The research meticulously characterizes the prevalence, nature, and impact of changes made to online news articles after their initial publication, highlighting a significant challenge to content integrity and public trust.

The core premise of the research is that these unannounced or "stealth" edits can lead to split worldviews, where readers accessing the same article at different times form fundamentally different understandings of an event or topic. This divergence contributes to widespread skepticism and a growing belief among the public that news organizations may intentionally mislead. The speakers argue that while the security community has long addressed issues like black hat SEO and domain name integrity, content integrity, particularly concerning post-publication changes, represents a burgeoning and critical area for research and intervention, extending beyond news to encompass all forms of user-generated content.

The work presented by Tsoukaladelis and his team is crucial because it provides the first large-scale, data-driven characterization of this pervasive issue. By analyzing hundreds of thousands of articles from diverse publishers, they quantify the frequency of changes, categorize their syntactic and semantic impact, and expose the alarming rate at which these alterations go unannounced. Their findings underscore the urgent need for standardized transparency mechanisms to restore trust and ensure that the historical record of online information remains accurate and accessible to all.

Background

▶ Watch: Introduction: The rise of online news and 'split worldviews' (0:00)

The evolution of news consumption from traditional print media to digital platforms has brought about a paradigm shift, offering numerous advantages such as enhanced portability, instant updates, and broader accessibility. In 1440, the invention of the Gutenberg press revolutionized information dissemination; five and a half centuries later, 60 million newspapers circulated daily in the US. Today, a mere three decades further, the vast majority of Americans primarily consume news online. This digital transformation has democratized access to diverse viewpoints and global news, while also lowering the overall cost for consumers and integrating various media formats like text and video.

However, this rapid digital transition has also introduced a host of concerning trends that erode the integrity and trustworthiness of news. Readers now typically only glance at article titles, with the average time spent per visit declining from 2.5 minutes in 2014 to just 1.5 minutes. The dissemination of news via social media incentivizes publishers to craft clickbait titles, prioritizing engagement over accuracy. More critically, the digital nature of online articles allows for stealth edits—post-publication changes made without alerting readers—a practice that was nearly impossible with physical newspapers.

This ability to silently alter content creates what the researchers term split worldviews. A reader encountering an article before a significant edit might form a completely different understanding of an event than someone reading the same article post-edit. The talk highlights a New York Times example where an original version and a later, edited version would lead to drastically different interpretations of an event. This phenomenon directly feeds into a broader crisis of trust: half of Americans now believe that national news organizations intend to mislead, misinform, or persuade the public.

The security community has historically focused on integrity issues such as black hat SEO, cloaking, residual trust, and domain name manipulation. This research argues for the inclusion of content integrity as an equally vital concern, extending its relevance beyond news articles to all forms of user-generated content, such as social media posts. The current primary mitigation strategy, third-party archiving services like the Wayback Machine, is a single point of failure and suffers from limitations such as paywalls, hindering comprehensive and timely content preservation. The motivation for this research was starkly illustrated by an article about Elon Musk, initially highly critical of his upbringing in South Africa, which was later significantly edited to adopt a more neutral tone, with the changes largely going unmentioned in any official correction. While a correction was later added, it referred to different changes, leaving the substantial alterations to the narrative unacknowledged, underscoring the subtle yet profound impact minute details can have on reader perception.

Key Findings

▶ Watch: Elon Musk article example of significant, uncorrected content changes (3:10)

The research provides compelling quantitative evidence regarding the prevalence and nature of post-publication changes to online news articles, revealing several critical insights:

• Prevalence of Edits: A significant 28% of all analyzed articles exhibited some form of post-publication edit. This demonstrates that content changes are not isolated incidents but a widespread phenomenon across online news.
• Syntactic Change Variability: The extent of syntactic changes varied considerably among publishers. For instance, OAN (One America News Network) had changes in approximately 15% of its articles, but when changes did occur, they were substantial, with a median edit distance of 25%. In contrast, the New York Times saw changes in about 40% of its articles, but these changes were generally smaller, with a median edit distance closer to 5%. This suggests different editing philosophies or practices across news organizations.
• Semantic Impact on Meaning: Of the paragraphs that underwent changes, a substantial 22% were found to not follow from the original paragraph, indicating that the meaning or core message of the content had been altered. This finding directly supports the "split worldview" hypothesis, as readers encountering the article before and after such an edit would derive different information.
• Semantic Impact on Sentiment: Beyond factual meaning, the sentiment of changed paragraphs was also affected. 7% of changed paragraphs exhibited an alteration in their sentiment, moving from positive to negative, neutral to negative, or vice-versa. This highlights how edits can subtly or overtly shift the emotional tone and persuasive power of an article.
• Prevalence of Stealth Edits: A particularly alarming finding is that 60% of changed articles did not provide any notification or correction regarding the edits. These unannounced changes, termed "stealth edits," are the primary drivers of the content integrity issue, as readers are left entirely unaware that the information they are consuming may have been revised.
• Lack of Standardization in Transparency: Publishers exhibit widely differing and inconsistent policies regarding how they inform readers of updates. Some may provide technical updates (e.g., "last modified" timestamps) without detailing changes, while others might offer proper updates (explicit corrections) but not for all alterations. The researchers noted that for some publishers, like the BBC, they were unable to find evidence of either technical or proper updates.

These findings collectively underscore a systemic challenge to content integrity in online news, demonstrating that post-publication changes are frequent, often substantial in their semantic impact, and predominantly occur without reader notification, contributing significantly to a crisis of trust and potential misinformation.

Technical Deep Dive

▶ Watch: Overview of data collection and syntactic/semantic analysis (4:10)

The research undertaken to characterize post-publication changes to online news employed a rigorous methodology, combining extensive data collection with advanced natural language processing (NLP) techniques. The core objective was to move beyond anecdotal evidence and provide a quantitative, systematic analysis of this phenomenon.

The experiment began with a large-scale data collection phase. To ensure a balanced and representative dataset, the researchers selected news publishers based on two key criteria:

1. Popularity: Utilizing traffic data from sources like Trang, they included publishers of varying popularity.
2. Political Leaning: Leveraging the AllSides Media Bias Chart, they ensured representation across the political spectrum, aiming to mitigate any inherent bias in their selection.

From these diverse sources, the team collected over 600,000 articles. Each article was crawled twice: first, immediately upon its initial publication, and second, several months later. This two-stage crawling process was crucial for capturing the state of the article at two distinct points in time, allowing for the identification of any intervening changes.

After collection, the raw HTML data for each article underwent a meticulous parsing and cleaning process. The goal was to isolate the core content of the article from dynamic elements that frequently change but are not part of the editorial content. This included removing advertisements, related news suggestions, user comments, and other dynamic widgets, ensuring that only the actual article text was retained for comparison.

The analysis was then systematically divided into two main categories: syntactic changes and semantic changes.

Syntactic Analysis

The syntactic analysis focused on the structural and surface-level alterations within an article. For this, the researchers primarily employed edit distance, specifically the Levenshtein distance, as their metric. Levenshtein distance quantifies the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one word or paragraph into another. This metric provided a high-level view of:

• Prevalence of change: How many articles had any change.
• Magnitude of change: The extent of alterations when they occurred. As mentioned in the key findings, OAN showed a 25% median edit distance when changes happened, indicating substantial alterations, while The New York Times had a lower 5% median edit distance despite more frequent changes.
• Topology of changes: Where in the article changes occurred (e.g., closer to the top, middle, or bottom).
• Length of changes: Measured in word levels, indicating how many words were added, removed, or modified.
• Paragraph additions/removals: Tracking the net change in paragraph count.

This initial syntactic pass identified 165,000 articles that exhibited some form of post-publication edit, generating hundreds of thousands of changed paragraphs for further, deeper analysis.

Semantic Analysis

Given the sheer volume of changed paragraphs, manual review for semantic changes (i.e., changes in meaning or sentiment) was impractical. The researchers therefore leveraged sophisticated NLP models for this task:

1. Entailment Analysis (Meaning Change):

• Tool: OpenAI GPT-3.
• Methodology: GPT-3 was used to analyze pairs of original and edited paragraphs to determine if the edited version logically followed from, contradicted, or was neutral towards the original. The focus was on identifying instances where the meaning of a paragraph had fundamentally shifted.
• Finding: This analysis revealed that 22% of changed paragraphs did not follow from the original, meaning their core message or factual content had been altered. This directly impacts the "split worldview" phenomenon.

1. Sentiment Analysis (Sentiment Change):

• Tool: A RoBERTa-based model.
• Methodology: This model was employed to assess the sentiment (e.g., positive, negative, neutral) of both the original and the post-edit versions of paragraphs. The goal was to identify cases where the emotional tone or bias of the content had changed.
• Finding: The sentiment analysis indicated that 7% of changed paragraphs also had their sentiment altered.
• Examples from the talk:
• Negative shift: An original quote, "That's really Theo of this Administration," was edited to include "He added that the Biden Administration is certainly behind the curve on many things not putting Americans first," clearly painting the paragraph in a more negative light.
• Neutral-to-dimmed shift: A paragraph originally stating Ukrainian President Volodymyr Zelenskyy received "a warm welcome from both sides of the aisle" was edited to drop the "receiving warm welcome" part and instead invoke comparisons to the September 11 attacks and Pearl Harbor, effectively diminishing the initial positive sentiment.

To gauge the accuracy of these NLP models, the researchers also performed manual sampling of the results, confirming the models' efficacy in identifying semantic shifts.

Characterizing Stealth Edits

Beyond identifying changes, the research meticulously categorized how publishers handle notifications of these changes, leading to a crucial classification:

• Proper Updates: These are explicit corrections, editor's notes, or updates where the publisher clearly states what changes were made, ideally explaining why, and informing the reader.
• Technical Updates: These involve a notification that an article has been modified, often through a "last modified on [date]" timestamp. However, the specific changes themselves or the reasons for them are not detailed.
• No Updates (Stealth Edits): This category encompasses changes made without any form of notification to the reader, rendering them completely unaware of the alteration. The study found that a staggering 60% of changed articles fell into this category.

The analysis also highlighted a significant lack of standardization in how publishers implement these update policies. Some publishers might consistently provide technical updates but rarely proper ones, while others might do the opposite. The researchers noted instances, such as with the BBC, where they found no evidence of either technical or proper updates for edited articles, underscoring the fragmented and inconsistent landscape of content integrity reporting. This technical framework allowed the researchers to move beyond qualitative observations to provide a robust, data-driven characterization of the problem, laying the groundwork for future solutions in content integrity.

Demo / Proof of Concept

▶ Watch: Using NLP (GPT-3, Roberta) for semantic analysis of changes (7:00)

The talk "The Times They Are A-Changin': Characterizing Post-Publication Changes to Online News" focused on presenting the methodology, findings, and implications of a comprehensive research study. While the presentation included specific examples of articles that had undergone significant changes, such as the Elon Musk piece and sentiment-altering quotes, these served as illustrative case studies derived from their extensive data analysis rather than a live demonstration of a specific tool or proof-of-concept software. The core contribution was the characterization framework and the empirical results, not a deployable system.

Defensive Implications

▶ Watch: Semantic results: Sentiment and meaning changes in paragraphs (7:55)

The findings from "The Times They Are A-Changin'" carry profound defensive implications for both news consumers and the broader information ecosystem. The pervasive nature of unannounced post-publication edits necessitates a fundamental shift in how we approach content integrity online.

For individual news consumers, the primary defense is heightened skepticism and a critical approach to information. Readers should be aware that the article they read today might not be the same as the one published yesterday. While difficult to implement at scale, cross-referencing information, seeking multiple sources, and being wary of sudden shifts in narrative or sentiment within an article could be initial steps. However, the burden should not solely fall on the consumer.

The research emphatically calls for a new standard for content integrity. This standard should mandate that news publishers transparently report all post-publication changes. This goes beyond simple "last modified" timestamps; it requires explicit, detailed proper updates that clearly state what was changed, why it was changed, and when. Potential mechanisms for such a standard could include:

• Blockchain-based integrity solutions: Distributing article hashes on a blockchain could provide an immutable, verifiable record of content at specific times, allowing readers or automated tools to detect any unauthorized or unannounced changes.
• Standardized API for change logs: Publishers could expose an API that provides a structured, machine-readable log of all revisions, enabling third-party tools (e.g., browser extensions, news aggregators) to highlight changes automatically.
• Industry-wide best practices: News organizations, perhaps through journalistic associations, could collaboratively develop and adopt a universal protocol for reporting edits, making transparency a core tenet of digital journalism.

Beyond news articles, the speakers highlight that user-generated content (UGC) faces the same pitfalls. Social media posts, blog entries, and online reviews can all be altered post-publication without notice, potentially manipulating public discourse or individual perceptions. Therefore, the proposed content integrity standard should be broad enough to encompass all forms of online content where trust and historical accuracy are critical.

Archiving services like the Wayback Machine, while valuable, are currently the only widespread mitigation. Defenders should recognize their limitations (single point of failure, paywalls, potential for delayed archiving) and advocate for more robust, decentralized, and accessible archiving solutions. Integration of content integrity checks directly into browser functionalities or news reading applications could empower users by automatically flagging edited content.

Ultimately, the defensive strategy must shift from reactive individual vigilance to proactive systemic transparency. News organizations, platform providers, and the security community must collaborate to develop and enforce mechanisms that ensure the integrity of online content, making stealth edits a relic of the past and fostering a more trustworthy digital information environment. This research opens a new direction in the integrity umbrella, underscoring its growing importance as misinformation and disinformation become increasingly prevalent.

Key Takeaways

• Prevalence of Edits: Approximately 28% of online news articles undergo post-publication changes, making this a widespread and significant challenge to content integrity.
• Semantic Impact: A substantial 22% of changed paragraphs alter the original meaning, and 7% also shift the sentiment, demonstrating that edits frequently impact the core message and tone of news.
• Pervasive Stealth Edits: A critical finding is that 60% of these changes are completely unannounced, leaving readers unaware that the information they are consuming has been modified.
• Split Worldviews: Unannounced edits contribute to "split worldviews," where readers accessing the same article at different times form fundamentally different understandings, eroding public trust in news organizations.
• Lack of Standardization: There is a severe lack of consistent and transparent policies among publishers regarding how they report post-publication changes, highlighting the need for industry-wide standardization.
• Call for New Integrity Standard: The research advocates for a new standard that informs users when online content, including news articles and user-generated content, has changed, moving beyond current limited archiving services to ensure greater transparency and content integrity.

About the Speaker(s)

The research presented in this talk, "The Times They Are A-Changin': Characterizing Post-Publication Changes to Online News," was a collaborative effort by Chris Tsoukaladelis, Brian Kondracki, Niranjan Balasubramanian, and Nick Nikiforakis. Chris Tsoukaladelis delivered the presentation at IEEE S&P. While specific titles and affiliations beyond their involvement in this paper are not detailed in the transcript, their collective work highlights expertise in cybersecurity, data analysis, and natural language processing, focusing on critical issues related to information integrity and online trust. Their contributions underscore a growing recognition within the security community of the importance of addressing content integrity in the digital age.