Building Onramps for Emergency Web Archiving in Ukraine and Beyond

Building Onramps for Emergency Web Archiving in Ukraine and Beyond

Speakers: Quinn Dombrowski, Academic Technology Specialist, Stanford

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=4uV61EHLKKA

Overview

In the wake of Russia's full-scale invasion of Ukraine in February 2022, a critical but often overlooked battlefront emerged: the preservation of Ukraine's digital cultural heritage. Quinn Dombrowski, an academic technology specialist at Stanford, delivered a compelling talk detailing the rapid, volunteer-driven efforts to archive Ukrainian websites and digital artifacts before they could be destroyed, altered, or lost amidst the conflict. This initiative, known as Saving Ukrainian Cultural Heritage Online (SUCHO), evolved from an urgent, decentralized response to a sophisticated, multi-faceted project addressing both digital and physical preservation needs.

Dombrowski's presentation highlights the unique challenges of emergency web archiving in a warzone, contrasting it with more traditional, planned data rescue operations. It underscores the fragility of digital information and the profound cultural significance embedded in online materials, from museum websites and academic databases to children's dance programs and internet memes. The talk ultimately serves as a powerful call to action, advocating for a widespread, community-led approach to digital preservation, recognizing that in an increasingly volatile world, the responsibility for safeguarding our shared heritage may fall to individuals and grassroots efforts rather than solely to institutions.

The work of SUCHO exemplifies how interdisciplinary skills, combining humanities scholarship with technical acumen, can be leveraged for immediate social impact. By mobilizing hundreds of volunteers globally, the project not only secured invaluable digital assets but also forged new methodologies and tools for rapid-response archiving. This talk is particularly relevant to the DEF CON audience, demonstrating how hacker ethos—resourcefulness, problem-solving, and a commitment to open-source solutions—can be applied to humanitarian crises, transforming passive "doomscrolling" into active, meaningful contributions to cultural resilience.

Background

▶ Watch: Introduction to speaker and digital humanities field (0:00)

Quinn Dombrowski's journey into emergency web archiving began from a background in digital humanities, a broad interdisciplinary field that applies computational methods to humanities research. Her prior work, including projects like the Data Sitters Club for computational text analysis, typically revolved around academic pursuits. However, a parallel thread within digital humanities, known as socially engaged digital humanities, emerged over the last decade, focusing on using technical skills to address social issues like government oppression, disaster relief, and data preservation. Examples include projects like Torn Apart/Zapatas, which tracked ISIS territory, and Nimble Tents Toolkit, a response to Hurricane Maria, and Data Refuge, which archived U.S. government data after the 2016 election. Despite these precedents, Dombrowski initially felt these issues were "not her lane."

The full-scale Russian invasion of Ukraine on February 24, 2022, dramatically shifted this perspective. For Dombrowski, whose academic background includes medieval Slavic studies and linguistics, the cultural dimension of the conflict—Russia's attempt to deny Ukrainian cultural identity and language—made the need for intervention undeniable. This personal connection, combined with her technical skills, eliminated all previous excuses not to get involved.

The impetus for SUCHO came quickly. On February 26, 2022, a music librarian named Anakius tweeted about organizing a virtual data rescue session for Ukrainian music collections. Sebastian Mastovich from Vienna responded, suggesting the involvement of Webrecorder and its open-source crawler, BrowserTricks, to facilitate a more distributed archiving approach beyond the centralized Internet Archive. This marked the genesis of SUCHO.

A critical challenge from the outset was the complete absence of a roadmap for such an emergency. Previous data rescue efforts, such as those during presidential transitions, operated under the assumption of stable infrastructure and predictable timelines. In contrast, the Ukrainian context involved websites potentially under active cyberattack, facing power outages, and network instability. The team faced profound uncertainty, not knowing how long critical sites would remain online or accessible. This immediate, high-stakes environment necessitated an agile, experimental approach to digital preservation, driven by the urgency that "every hour counted."

Key Findings

▶ Watch: Speaker's personal call to action after Ukraine invasion (4:00)

The SUCHO project yielded several crucial insights into the nature of emergency web archiving and community mobilization. One of the earliest and most impactful findings was the vulnerability of centralized archiving infrastructure. Just days after SUCHO launched, on March 4th, 2022, the Internet Archive (IA) experienced a power outage, rendering its Wayback Machine inaccessible for several hours. This event, caused by "our extremely janky Northern California power company" rather than Russian attacks, underscored the critical need for a distributed archiving model. It forced volunteers to consider alternatives and take a "technical leap" to learn new, more complex tools, ensuring that power outages in one region would not halt global preservation efforts.

Another significant discovery was the immense power of volunteerism and the accessibility of complex technical tasks when supported by a dedicated community. Within a day of launching, SUCHO attracted 419 people to its Slack instance, growing to 400 volunteers by March 2nd. These volunteers spanned a wide spectrum of technical proficiency, from seasoned tech professionals to individuals with no prior command-line experience, and even children and retirees. The project successfully onboarded these diverse groups, teaching them how to install Docker and run web archiving software, demonstrating that a "learning how to learn" ethos, similar to that found within the hacker community, was the most crucial skill. This inclusive approach led to volunteers dedicating personal resources, like hard drive space and old laptops, to the cause, and even resulted in innovative solutions like setting up BrowserTricks on a Raspberry Pi.

The project also revealed that digital cultural heritage, while vital, represents only a "tiny fraction" of Ukraine's overall heritage. Initially focused solely on virtual preservation, SUCHO received persistent feedback from Ukrainian libraries, archives, and museums about their desperate need to digitize physical materials. This led the project to expand its scope, securing grants (e.g., from the Pearl Jam Vitalogy Foundation) to supply scanners and, crucially, power banks. The latter became equally or more important, allowing libraries to function as community centers for phone charging during power grid attacks. This demonstrated the interconnectedness of digital and physical preservation and the need for flexible, responsive aid during crises.

Finally, the long-term sustainability of emergency archiving efforts presented a fundamental finding, encapsulated in the "nimble tents and bunkers" metaphor. While rapid, decentralized "Molotov cocktail" efforts by volunteers (the "nimble tents") are crucial for immediate response, they are inherently temporary as volunteers eventually "move on with their lives." The ideal scenario, per Dombrowski, is for national libraries and governments to act as "bunkers" for long-term data storage. However, the talk critically notes that in current geopolitical climates (e.g., in the U.S.), institutions and governments "may in fact be more of a liability than help." This leads to the sobering conclusion that the future of web archiving might increasingly rely on individuals and communities building their own "tents," learning DIY methods, and distributing data across personal storage devices, highlighting a shift towards extreme decentralization out of necessity.

Technical Deep Dive

▶ Watch: Understanding the cultural heritage aspect of the conflict (4:50)

The SUCHO project leveraged a diverse array of tools and methodologies to address the complex challenges of emergency web archiving. Initially, the project relied on accessible, familiar, and often free tools to quickly mobilize volunteers. A Slack instance served as the primary communication hub, facilitating rapid coordination among hundreds of geographically dispersed participants. A massive Google Sheets spreadsheet became the central repository for tracking websites, their status, and volunteer assignments, which at times saw "several hundred people editing... at the same time." For less technically proficient volunteers, the Internet Archive's Wayback Machine was the primary tool for submitting links.

As the project scaled and the limitations of centralized solutions became apparent, SUCHO pivoted towards a more distributed, open-source approach. The core of this strategy involved Webrecorder tools, particularly the BrowserTricks open-source web archiving crawler. This allowed individual volunteers to perform high-fidelity crawls locally. To overcome technical barriers for non-developers, SUCHO conducted Zoom sessions, providing "hand-holding" instructions for installing Docker and running Webrecorder software even for those who had "never seen the command line before." This training enabled a wide range of devices, including old laptops and even a Raspberry Pi, to contribute to the crawling effort.

The archiving workflow was structured in several phases:

1. Discovery: Initial lists were generated from user submissions, Wikidata, domain intelligence, and map-based Open Source Intelligence (OSINT). Slavic librarians provided expert curation. In a novel application of OSINT, volunteers would monitor missile warnings, then use Google Maps to virtually "walk through the streets of cities under fire," identifying cultural heritage sites and checking for associated websites not yet in their spreadsheet.
2. Prioritization: Sites were prioritized based on cultural importance and immediate risk. An initial attempt to prioritize based on IP address geolocation (assuming servers within Ukraine were most vulnerable to physical attack) was quickly abandoned as it became clear that externally hosted sites were equally at risk due to refugees' inability to pay hosting bills.
3. Triage: Technical feasibility was assessed, determining how easy a site was to crawl and if it required specialized scraping techniques. This was particularly critical for complex sites like 3D virtual museum walkthroughs, which required dedicated volunteers to master their unique archiving challenges. Old, proprietary Russian library catalog software, for instance, required a graduate student to "read through like this dense PDF manual" to reverse engineer it for scraping.
4. Crawling: A combination of the Wayback Machine and the full suite of Webrecorder tools was employed.
5. Quality Control: Multilingual volunteers reviewed archived materials to ensure comprehensive and accurate capture.

Beyond raw archiving, SUCHO developed several public-facing technical resources. The project established gallery.sucho.org using the open-source Omeka software, providing an infrastructure for Ukrainian institutions to host digitized images when they lacked their own. The interface was fully translated into Ukrainian, marking its first availability in the language. A particularly innovative technical achievement is memes.sucho.org, a mobile-friendly "meme wall" currently housing over 8,000 annotated and translated Ukrainian war memes. This resource, led by Anna Rakanskaya and Simon Wilds, serves as a crucial historical record of the war's narrative as experienced through popular culture. The project also released database.sucho.org, an audited database of all created web archives, linking metadata to their origins and creators, a cleanup effort that took "months and months and months to do."

For data storage, SUCHO initially used S3 buckets but later migrated to Wasabi, which committed to "at least 5 years of storage." The project faced the challenge of long-term data custodianship, with an initial hope to transfer data to a UNESCO-facilitated national digital library of Ukraine that "never really got past the planning stage." This led to a decentralized storage model, with a "loose network of folks," particularly in the EU (e.g., Germany), hosting copies of the data, acknowledging the logistical downsides but emphasizing its perceived safety in an uncertain future.

Demo / Proof of Concept

▶ Watch: The tweet that initiated the archiving effort (6:00)

While the DEF CON talk itself did not feature a live, interactive technical demonstration, the entire SUCHO initiative serves as an expansive and highly successful proof of concept for rapid, decentralized, and community-driven emergency web archiving. The project's existence and its tangible outputs demonstrate the viability of its core technical and organizational strategies.

The most compelling "demonstration" of SUCHO's capabilities lies in the sheer volume and diversity of cultural heritage it successfully preserved. The project archived a vast array of digital content, from 18th-century music manuscripts and websites of the National Museum of Folk Architecture and Life (featuring vulnerable thatched-roof buildings) to virtual museum walkthroughs and even local after-school dance programs for children. This breadth of captured data, as highlighted by Dombrowski, showcases the project's ability to adapt its technical approach to various digital formats and content types.

Specific platforms developed by SUCHO further exemplify its technical achievements:

• gallery.sucho.org: This platform, built on Omeka with a fully translated Ukrainian interface, demonstrates SUCHO's commitment to creating accessible infrastructure for Ukrainian cultural heritage institutions. It proves that open-source solutions can be rapidly deployed and localized to meet specific in-country needs.
• memes.sucho.org: The "meme wall" is a groundbreaking proof of concept for archiving ephemeral, culturally significant internet content. Its mobile-friendly design, combined with the successful collection, annotation, and translation of over 8,000 memes, showcases the technical feasibility of preserving dynamic digital folklore for future historians and researchers. This is a direct response to the understanding that "memes of today are cultural heritage."
• database.sucho.org: This comprehensive, audited database of all archived materials stands as a testament to the project's commitment to metadata integrity and discoverability. It proves that even in a chaotic, emergency context, meticulous record-keeping and data organization are achievable, providing a structured catalog of their distributed efforts.

The rapid mobilization of over 400 volunteers, many without prior technical experience, and their successful training in using tools like Docker and BrowserTricks on personal devices (including Raspberry Pi), is itself a powerful proof of concept for large-scale, distributed technical education and execution. It validated the idea that technical skills, when broken down and supported, can be effectively disseminated for humanitarian purposes, turning "doomscrolling" into active contribution. The project successfully demonstrated that "people can respond very quickly and do things" without being an "official institutionally sanctioned initiative," proving the efficacy of a "Molotov cocktail" approach to emergency response.

Defensive Implications

▶ Watch: Introducing Webrecorder for distributed web archiving (6:30)

The SUCHO project offers profound defensive implications for individuals, communities, and institutions grappling with the increasing ephemerality of digital information and the threats posed by conflict and political instability. The core message is a proactive call to action: "you need to be the person who does this web archiving."

1. Embrace Decentralized and DIY Archiving: The experience with the Internet Archive's power outage and the broader geopolitical landscape (where governments and large institutions "may in fact be more of a liability than help") strongly advocates for a decentralized model. Defenders should learn and implement DIY web archiving techniques using open-source tools like Webrecorder and BrowserTricks. The project's successful training of non-technical volunteers in Docker indicates that these skills are learnable for a broad audience. The advice to "store it saved on flash drives, make a copy of the flash drive, mail the flash drive to a friend" encapsulates this shift toward personal, distributed resilience.

1. Anticipate and Mitigate Infrastructure Failure: The talk underscores that threats extend beyond direct attacks. Power grid failures, as experienced in Ukraine, and even mundane issues like unpaid hosting bills (for refugees) can lead to data loss. Defenders need to plan for these contingencies, ensuring data is mirrored geographically and that hosting solutions are robust or diversified. The SUCHO team's quick pivot to distributed archiving after the IA outage is a prime example of adapting to unforeseen infrastructure vulnerabilities.

1. Prioritize Cultural Heritage in Crisis Planning: The project highlights that cultural heritage is often a target in conflicts, not just for physical destruction but also for erasure or reappropriation of narratives. Security planning should explicitly include strategies for preserving digital cultural assets, recognizing their role in national identity and historical record. This extends beyond high-value government data to include everyday cultural touchstones, like local community websites.

1. Leverage Open Source Intelligence (OSINT) for Proactive Archiving: SUCHO's innovative use of OSINT, monitoring missile warnings and virtually exploring Google Maps for cultural sites, demonstrates a proactive defensive posture. Security professionals and archivists can adapt similar methodologies to identify at-risk digital assets in real-time during crises, moving beyond reactive preservation.

1. Cultivate Interdisciplinary Collaboration: The success of SUCHO, bridging digital humanities with technical communities, emphasizes the power of interdisciplinary collaboration. Defenders should actively seek partnerships with librarians, historians, and cultural heritage experts who possess domain knowledge to identify critical data, while technologists provide the means for preservation. This broadens the scope of what is considered "defensible" and brings diverse perspectives to problem-solving.

1. Recognize the Ephemeral Nature of All Websites: Dombrowski's assertion that "there's nothing more ephemeral than a website," even within stable governments (citing U.S. government data being "deleted and destroyed quietly"), serves as a stark warning. The defensive posture should be one of continuous vigilance, recognizing that all online content, regardless of its origin, is subject to deletion or alteration. This necessitates ongoing archiving efforts, not just during overt crises.

Key Takeaways

• Emergency web archiving is a critical, immediate need in times of conflict and instability, often requiring rapid, decentralized responses. The SUCHO project launched within days of the 2022 Ukraine invasion, driven by the urgency that "every hour counted."
• Centralized archiving solutions are vulnerable, necessitating distributed and open-source approaches for resilience. The Internet Archive's outage highlighted the importance of tools like Webrecorder and BrowserTricks for creating multiple, independent copies of data.
• Technical skills for web archiving can be taught to a broad, non-technical audience, fostering widespread community participation. SUCHO successfully trained hundreds of volunteers, from children to retirees, in using Docker and command-line tools.
• Cultural heritage extends beyond traditional artifacts to include diverse digital content, from museum websites to internet memes. The project's archiving of 3D walkthroughs, old library catalogs, and over 8,000 annotated memes on memes.sucho.org demonstrates this breadth.
• Emergency digital preservation efforts often reveal needs for physical preservation and infrastructure support. SUCHO expanded to provide scanners and crucial power banks to Ukrainian libraries, recognizing the interconnectedness of digital and physical resilience.
• The future of web archiving may increasingly rely on DIY, individual, and community-led efforts, as institutional support can be slow or unreliable. The "nimble tents and bunkers" metaphor emphasizes the need for individuals to take personal responsibility for preserving data they care about.

About the Speaker(s)

Quinn Dombrowski is an Academic Technology Specialist at Stanford University, working across the Department of Non-English Literatures and the University Library. Her professional background is rooted in the interdisciplinary field of digital humanities, where she applies computational methods to humanities research. Dombrowski holds a BA and MA in medieval Slavic and comparative Slavic linguistics from the University of Chicago. Initially pursuing a PhD, she pivoted towards a career focused on helping others solve problems and conduct research more effectively using technology.

Prior to her role at Stanford, Dombrowski gained experience supporting high-performance computing clusters at UC Berkeley and worked on Mellon-funded grants for digital humanities cyberinfrastructure. Her expertise combines deep knowledge of Slavic languages and cultures with practical technical skills. This unique blend made her uniquely positioned to lead the SUCHO initiative, leveraging her academic background, technical acumen, and passion for socially engaged digital work to address the urgent crisis of cultural heritage preservation in Ukraine. Her work exemplifies how humanities scholars can transition into critical technical roles, applying their understanding of culture and language to pressing global challenges.