Who Watches the NPM Watchers?

Who Watches the NPM Watchers?

Speakers: Paul McCarty, Founder and Creator, Open-Source Malware

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=AcLBRqmj4lI

Overview

In his thought-provoking BSides SF talk, "Who Watches the NPM Watchers?", Paul McCarty, co-founder of Open-Source Malware, delves into the critical, yet often unexamined, landscape of NPM package scanning. The presentation uncovers the varying methodologies, motivations, and significant blind spots of organizations tasked with monitoring the world's largest software registry. McCarty’s research employs an innovative approach using "canary packages" to observe and fingerprint the entities — from cloud providers and security vendors to nation-state actors — that are actively scrutinizing the NPM ecosystem.

This talk is particularly pertinent given the ever-increasing reliance on open-source software and the escalating threat of supply chain attacks. NPM, with its nearly 4 million packages and billions of weekly downloads, represents an unparalleled attack surface. McCarty's work highlights that despite its critical infrastructure status and ownership by a tech giant like Microsoft, NPM's self-scanning capabilities are severely lacking. The research serves as a stark reminder that the security of our software supply chain is not uniformly protected and reveals critical gaps that adversaries can and do exploit.

McCarty's objective is not merely to expose vulnerabilities but to foster a deeper understanding of the "watchers" themselves. By meticulously tracking who scans, what they prioritize, and how efficiently they operate, the talk provides invaluable intelligence for both defenders seeking to harden their systems and developers aiming for more secure practices. It underscores the urgent need for a more comprehensive and coordinated approach to open-source software security, moving beyond assumptions to data-driven insights into the actual state of package monitoring.

Background

▶ Watch: Introduction to speaker Paul McCarthy and research background (0:00)

The problem addressed by McCarty's research is rooted in the colossal scale and inherent architectural vulnerabilities of the NPM (Node Package Manager) ecosystem. As the world's largest software registry, NPM hosts almost 4 million packages, experiences 140,000 updates daily, and sees 3,800 new packages added every day – a threefold increase in new packages over just two years. This explosion of content fuels an astounding 20 billion weekly downloads, making every package a potential vector for attack. Despite being owned by Microsoft, a highly capitalized organization, McCarty critically notes that NPM's security posture remains a significant concern, often described as a "crappy show."

Several factors contribute to this precarious situation. A fundamental issue is that much of the metadata associated with NPM packages can be easily faked. While NPM does link a verified email address to a user account (a safeguard PyPI notably lacks), malicious actors can simply create new accounts, bypassing this minimal barrier. More critically, the design philosophy of JavaScript—a language built with "no batteries included"—necessitates importing virtually everything from external libraries. This has led to an unprecedented level of transitive dependencies. NPM packages average around 680 to 700 transitive dependencies, an order of magnitude higher than other languages like PHP, which sits at roughly 68 to 70. This creates an enormous, complex, and often opaque attack surface where a vulnerability or malicious insertion deep within the dependency chain can compromise countless applications.

Furthermore, highly popular packages, such as debug (the fourth or fifth most downloaded package with 380 million downloads), are often maintained by small groups of individuals. These maintainers are susceptible to account takeover attacks through phishing, which has become a prevalent vector for injecting malicious code into widely used packages. Geopolitical considerations also play a role, with five of the eight major NPM mirrors globally located in China, raising concerns about potential manipulation and data hoarding, a topic McCarty explored in a previous talk titled "Panda Mirror."

McCarty's motivation for this research stems from his prior experience in red teaming for software supply chain security and his work with Open-Source Malware. He observed significant gaps in what security vendors were scanning for and how they were doing it. The core questions driving his inquiry were: Is NPM adequately scanning itself? What are security vendors prioritizing? And are criminals learning from these scanning patterns? The broader context also includes the pervasive issue of developers accidentally embedding sensitive credentials or data within NPM packages during publication, a problem exacerbated by GitHub's failure to implement its promised secret scanning program for NPM, despite a public announcement in 2023. This confluence of factors creates a challenging environment where the security of the software supply chain is far from assured, necessitating a deeper understanding of those who claim to monitor it.

Key Findings

▶ Watch: NPM's massive scale and inherent security vulnerabilities (2:00)

Paul McCarty's research yielded several critical findings that shed light on the fragmented and often insufficient state of NPM package scanning:

1. NPM's Inadequate Self-Scanning: Despite being owned by Microsoft, a global tech giant, NPM itself performs a "crappy job" of scanning its own packages. It relies heavily on the broader security researcher community to identify and report malicious packages, which are typically submitted to OSV (Open Source Vulnerabilities) or GHSA (GitHub Security Advisory). An internal Microsoft service does scan NPM, but it identifies a "very, very small number" of malicious instances, indicating a significant lack of proactive internal security investment by Microsoft for its NPM asset.

1. Varied Cloud Provider Scanning:

• Google: Scans from its Council Bluffs, Iowa data center. OSV also utilizes an automated scanner running out of Google, known as the OSV scanner.
• Microsoft Azure: Scans from various IPs in Virginia.
• AWS: Scans "every new NPM package" within five minutes from multiple US locations and occasionally India. Crucially, a specific team at AWS, led by Chai Tran, actively and consistently submits a "ton of malicious package submissions to OSV," demonstrating a strong commitment to public security.
• Alibaba, Tencent Volcano Engine (ByteDance), Huawei: These Chinese cloud providers engage in extensive, "mass scanning" of "every single NPM package for everything all the time" from numerous locations across China. Their operations often use older systems (e.g., a CentOS 7 2018 user agent string) and exhibit poor operational security, allowing for clear fingerprinting.
• Russia (VimpelCom): Scans originate from ASNs in Moscow and Tula, typically using the VimpelCom ISP. These scans are slower than those from US or Chinese providers, often taking 2 to 10 hours, sometimes even a full day, to process new packages.

1. Security Vendor Agendas and Coverage Gaps:

• ReversingLabs: Has scanned NPM for a long time but focuses almost exclusively on AWS credentials. They perform frequent scans (three to four daily) on packages, even when nothing has changed, indicating potential compute waste and a narrow scope of interest.
• Kaspersky: Known for its strong research team, Kaspersky's scanning operations originate from Russia (and a recent Swiss location). They scan most packages within 10-12 minutes but primarily target Word and Excel files, occasionally JavaScript payloads. They are more efficient, scanning only one time per release, but their IPs have been associated with mass reconnaissance scanning (e.g., WordPress, Joomla), suggesting a broader, less specialized recon effort.

1. Prevalence of Static Analysis vs. Dynamic Sandboxing: Most security companies (e.g., Ox Security, Socket, Endor) rely on static analysis to scan NPM packages. This approach, while efficient, often fails to detect multi-stage payloads or dynamically loaded malicious content, as it doesn't involve actually running the package. Conversely, the cloud providers and nation-state actors observed in McCarty's research (AWS, Azure, Chinese companies) are employing dynamic sandboxing, which allows them to observe runtime behavior and potentially uncover more sophisticated attacks.

1. GitHub's Unfulfilled Secret Scanning Promise: Despite a public announcement in 2023 that GitHub would begin scanning all NPM packages for embedded credentials using its existing secret scanning program, McCarty's research indicates this has not been implemented. This leaves a significant vulnerability, as developers frequently accidentally publish sensitive files, credentials, and even entire databases to NPM, as exemplified by a case involving a Middle Eastern visa company exposing passport photos and database credentials.

These findings collectively paint a picture of an NPM ecosystem where monitoring is inconsistent, driven by varied commercial or national interests, and often lacks the comprehensive dynamic analysis necessary to counter evolving supply chain threats. The reliance on external researchers, the narrow focus of some vendors, and the unaddressed issues of credential exposure underscore the urgent need for systemic improvements.

Technical Deep Dive

▶ Watch: The critical issue of extreme transitive dependencies (4:00)

Paul McCarty's research methodology hinges on the ingenious use of canary packages to observe and fingerprint entities scanning the NPM registry. A canary package is essentially a specially crafted NPM package embedded with one or more canary tokens. These tokens, primarily sourced from Thinkst Canary, are designed to trigger an alert when accessed, providing the researcher with details about the accessing entity, such as IP address, user agent, and time of access. McCarty collaborated with Thinkst Canary, leveraging their enterprise console for this research.

McCarty deployed a diverse set of eight common canary package types to capture a broad spectrum of scanning behaviors:

1. JavaScript Payload: This type includes a small JavaScript snippet configured to run automatically upon package installation or, in some cases, importation. The payload is designed to capture non-sensitive information, such as the host's public IP address and hostname, aligning with typical bug bounty program rules to prove execution without causing harm. User agent strings and other metadata are often captured automatically by the canary token endpoint.
2. Binary (EXE): An executable file was placed within the NPM package. Crucially, this binary was not set to run automatically, requiring explicit execution by a scanner to trigger the canary token. This tests for deeper analysis capabilities beyond basic static checks.
3. Word and Excel Files: Similar to the binary, these document files containing canary tokens were embedded but not auto-executing, requiring an opening action to trigger.
4. AWS Credentials: Specific canary tokens designed to detect access to simulated AWS credential files were used. This is a common target for attackers and therefore a relevant focus for security scanners.
5. DNS-based Canaries: These tokens leverage DNS lookups, triggering when a resolver attempts to resolve a unique canary domain, indicating package analysis.
6. Local Credential Files (.env): Files like .env are frequently and accidentally included in NPM packages. Canaries mimicking these files were used to detect scanners looking for such exposures.
7. PDFs: PDF documents with embedded canary tokens.
8. Web Images: A particularly clever technique involved embedding web images (e.g., <img> tags pointing to a canary URL) directly into the README.md file of the NPM package. This ensures that any entity simply opening or rendering the README.md (e.g., a web crawler or a security scanner previewing the package) would trigger the canary.

Upon deployment, the triggers from these canary packages were fed into a custom-built visualization tool, affectionately termed a "pew pew map." This map allowed McCarty to track the geographical origin and organizational affiliation of scanning activities in near real-time, providing a high-level overview of the "who" and "where" of NPM monitoring. McCarty emphasized that the canary packages were carefully designed to avoid triggering on casual npm install operations by individual users, focusing solely on organizational and automated scanning entities.

A significant technical distinction highlighted by McCarty is between static analysis and dynamic sandboxing.

• Static analysis involves examining the code without executing it. Many modern Software Composition Analysis (SCA) companies (e.g., Ox Security, Socket, Endor) primarily use this method. While efficient for identifying known vulnerabilities or patterns, it often fails to detect sophisticated multi-stage attacks where malicious code fetches subsequent payloads at runtime. McCarty notes that these static-only scanners were generally not present in his data, as they wouldn't trigger the dynamic canary tokens.
• Dynamic sandboxing, conversely, involves executing the package in a controlled, isolated environment and observing its behavior. Organizations like AWS, Microsoft Azure, and the various Chinese cloud providers were found to be utilizing dynamic sandboxing, as evidenced by their triggers on JavaScript payloads and other dynamic canaries. This approach is superior for detecting obfuscated or multi-stage malware, as it allows observation of the full kill chain, including fetches of second or third-stage loaders. McCarty points out that if sandboxes are run in a "closed loop" without internet access, even dynamic analysis can miss external payload fetches.

The data collected through these canaries allowed McCarty to fingerprint scanning organizations. For instance, Kaspersky was observed scanning from three locations in Russia and a new Swiss location, showing interest primarily in Word and Excel files, and scanning only once per release – an efficient use of compute. Their IPs were also linked to broader mass reconnaissance scanning campaigns on platforms like Shodan and VirusTotal. ReversingLabs was characterized by its almost exclusive focus on AWS credentials and its inefficient practice of repeatedly scanning unchanged packages daily. Chinese scanners were identified by their mass-scanning behavior, originating from numerous cloud providers (Ali Yun, Tencent, Huawei, ByteDance's Volcano Engine), often revealing outdated operating systems (e.g., CentOS 7 2018) in their user agent strings, indicating poor operational security and allowing for persistent tracking of their assets. This detailed fingerprinting enables a deeper understanding of each scanner's intent, capabilities, and, crucially, their blind spots.

Demo / Proof of Concept

▶ Watch: Revealing the geographic distribution of NPM mirrors (5:30)

During the talk, Paul McCarty referenced a live "pew pew map" that visualizes the triggers from his canary packages in real-time. He mentioned having it open and the possibility of demonstrating it at the end of the session if time permitted. While a detailed live demonstration of the map's functionality or the triggering of a canary package was not explicitly described in the transcript, the existence of this tool is central to his research methodology, allowing for the real-time collection and visualization of scanning activities.

Defensive Implications

▶ Watch: Introducing the 'canary package' research methodology (6:50)

The detailed insights from "Who Watches the NPM Watchers?" offer critical implications for various stakeholders in the software supply chain:

1. For NPM (Microsoft): The most pressing implication is the urgent need for NPM to significantly enhance its internal scanning capabilities. Relying on the broader security research community is not a sustainable or robust strategy for a critical infrastructure component. Microsoft should invest in comprehensive, proactive, and dynamic scanning of all NPM packages, moving beyond its currently minimal efforts. Furthermore, the unfulfilled promise of GitHub's secret scanning program for NPM must be addressed immediately to prevent the accidental exposure of sensitive credentials and data.

1. For Developers and Organizations Consuming NPM Packages:

• "Trust but Verify": Developers cannot blindly trust NPM packages. They must adopt a "trust but verify" mindset, actively analyzing what packages do, not just what they claim to do. This includes understanding the full scope of transitive dependencies, which often introduce the vast majority of the attack surface.
• Secure Publishing Practices: Implement rigorous checks to prevent the accidental inclusion of sensitive files (e.g., .env, configuration files, database backups, private keys) during npm publish. Tools that scan for credentials pre-publish should be utilized.
• Leverage Security Features: Actively use NPM's new security features, such as trusted publishing and mandatory MFA (Multi-Factor Authentication) for publish events, to protect against account takeover attacks.
• Understand Tool Limitations: Recognize that traditional SCA (Software Composition Analysis) tools are primarily designed to find accidental vulnerabilities, not intentionally malicious code or malware. Relying solely on SCA for malware detection leaves significant gaps.
• Consider Dynamic Analysis: Organizations should consider implementing or utilizing tools that perform dynamic sandboxing of NPM packages, especially those with high impact or unknown provenance. Static analysis alone is insufficient to detect multi-stage loaders or runtime malicious behavior.
• Software Firewalls with Caution: While tools like PNPM offer more secure defaults, and developer firewalls (e.g., Socket) aim to prevent risky package installations, these wrappers can be easily bypassed by determined developers if not strictly enforced and made responsive to legitimate needs. Their theoretical benefit must be weighed against practical bypass risks.

1. For Security Vendors:

• Address Coverage Gaps: Vendors must broaden their scanning scope beyond narrow agendas. The findings show that some vendors (e.g., ReversingLabs focusing only on AWS credentials, Kaspersky on Word/Excel) are missing other critical threat vectors. A comprehensive approach is necessary.
• Optimize Resource Usage: Inefficient scanning practices, such as repeatedly scanning unchanged packages (as observed with ReversingLabs), waste significant compute resources. Vendors should optimize their scanning logic to be more efficient and impactful.
• Embrace Dynamic Sandboxing: To detect sophisticated, multi-stage attacks, security vendors should increasingly adopt and improve dynamic sandboxing capabilities, ensuring sandboxes allow for observation of external payload fetches if that is part of the attack chain.

1. For Threat Intelligence and Security Research:

• Track the Watchers: The ability to fingerprint and track scanning organizations provides invaluable threat intelligence. Understanding who is scanning, what their priorities are, and their operational security (or lack thereof) can inform defensive strategies and help anticipate adversary movements.
• Identify Evasion Techniques: By understanding the coverage gaps of various scanners, defenders can better predict how adversaries might attempt to evade detection. This knowledge can be used to develop more robust detection mechanisms.

In essence, the talk calls for a paradigm shift from assuming security to actively verifying it, urging all stakeholders to be more intentional, comprehensive, and data-driven in their approach to NPM security.

Key Takeaways

• The watchers can be watched: It is possible to track and fingerprint organizations scanning NPM packages, revealing their intent, methodologies, and operational security posture. This intelligence is crucial for understanding the real-world threat landscape.
• Significant coverage gaps exist: Security vendors and cloud providers often have narrow scanning agendas (e.g., AWS credentials, Word/Excel files), leading to critical blind spots that adversaries can exploit. No single entity provides comprehensive coverage.
• NPM's self-scanning is insufficient: Microsoft, as the owner of NPM, is not adequately investing in proactive security scanning, largely relying on external researchers to identify malicious packages. GitHub's promised secret scanning for NPM has also not materialized.
• Static analysis is often inadequate: Most security companies rely on static analysis, which can miss sophisticated, multi-stage attacks that involve dynamic loading of payloads. Dynamic sandboxing is necessary to observe the full kill chain.
• Resource waste is common: Some scanning organizations exhibit inefficient practices, such as repeatedly scanning unchanged packages, leading to unnecessary compute waste.
• "Trust but verify" is paramount: Developers and organizations must adopt a skeptical approach to NPM packages, actively analyzing their behavior and dependencies rather than blindly trusting them, especially given the high volume of transitive dependencies and potential for accidental credential exposure.

About the Speaker(s)

Paul McCarty is a prominent figure in the field of open-source software security. He is one of the two co-founders and creators of Open-Source Malware, which he describes as the world's largest database of open-source malicious packages. McCarty is well-known as an NPM security researcher and also conducts significant research in the GitHub ecosystem. His interest in this area stemmed from past experiences building Secure Stack, where customers were compromised by malicious packages due to a lack of protective measures at the time.

Beyond his current work, McCarty has a background in various startups and has also contributed to larger organizations such as Boeing and JPL, as well as working in the "beltway" (referring to government or defense contracting work) in the 2000s. He is recognized on platforms like LinkedIn for his security research. McCarty has delivered several talks related to software supply chain security, including presentations on malicious packages bypassing security tools (DEF CON Adversary Village), extracting threat intelligence from supply chain attacks (Berlin CTI), and how Chinese entities manipulate NPM to hoard malware ("Panda Mirror" at BSides Canberra). He is also known by his handle "6mile."