Patchy Performance? Uncovering the Vulnerability Management Practices of IoT-Centric Vendors

Patchy Performance? Uncovering the Vulnerability Management Practices of IoT-Centric Vendors

Speakers: Sandra Rivera Pérez, Researcher, Delft University of Technology; Michel van Eeten, Professor, Delft University of Technology; Carlos H. Gañán, Researcher, Delft University of Technology

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=7JdKc_S2igo

Overview

This talk, presented by Sandra Rivera Pérez from Delft University of Technology, delves into a systematic analysis of the vulnerability management practices of vendors heavily invested in the Internet of Things (IoT) sector. Collaborating with Michel van Eeten and Carlos H. Gañán, and supported by the Intersect project, the research addresses a critical concern: as smart devices are projected to outnumber traditional ones by 3 to 1 by 2025, the security performance of IoT products becomes paramount. The study specifically investigates whether IoT-centric vendors—those predominantly focused on IoT—exhibit different security behaviors compared to their non-IoT-centric counterparts.

The core motivation for this research stems from widespread criticisms regarding the perceived inadequacy of security in IoT devices. While such concerns are prevalent, a systematic, empirical basis for these claims has often been lacking. The researchers sought to provide this foundation by quantifying the degree to which vendors are IoT-centric and then evaluating their preventive (number of vulnerabilities) and reactive (patch availability and timeliness) security performance. The findings challenge several common assumptions, offering a nuanced perspective on the complex dynamics of vulnerability management within the rapidly expanding IoT landscape.

The article will explore the methodology employed, the significant statistical findings, and the implications for both vendors and security defenders. By dissecting factors such as vendor size, the presence of vulnerability disclosure policies (VDPs), and bug bounty programs, the research sheds light on how these elements influence the discovery, reporting, and remediation of vulnerabilities. Ultimately, the study aims to inform current and future policies, providing an evidence-based rationale for interventions targeting IoT security.

Background

▶ Watch: Study's aims and key research questions (1:28)

The proliferation of IoT devices represents one of the most significant technological shifts of our era. Projections indicate that by 2025, smart devices will outnumber traditional computing devices by a ratio of 3:1, fundamentally reshaping our homes, industries, and critical infrastructure. This interconnected future, while promising immense convenience and efficiency, simultaneously introduces a vast and complex attack surface. The security implications of billions of new, internet-connected devices, many with limited computational resources and extended lifespans, are profound. Concerns about the security and privacy of these products are widespread, with many stakeholders fearing that IoT vendors are not adequately addressing these challenges.

Despite these growing concerns, a comprehensive, systematic analysis of the security performance of IoT vendors has historically been scarce. Prior research has often highlighted individual vulnerabilities or specific product categories, but a broader, empirical comparison between vendors deeply invested in IoT and those for whom IoT is a minor part of their portfolio has been missing. This gap in understanding perpetuates assumptions—often negative—about IoT security without sufficient data to support or refute them. The problem, therefore, is a lack of an empirical basis to understand the true state of vulnerability management in the IoT sector, making it difficult to formulate effective policies or defensive strategies.

To address this, the presented research built upon prior work by first identifying a set of 104 vendors deemed important in the market. A crucial initial step was to quantify the "IoT-centricity" of each vendor, determining whether their product portfolio predominantly focused on IoT or if IoT represented only a minor fraction. This distinction was vital for making meaningful comparisons. The study then aimed to answer three fundamental research questions:

1. Are IoT-centric vendors associated with more vulnerabilities than non-IoT-centric vendors?
2. Are IoT-centric vendors more or less likely to have patches available than non-IoT-centric vendors?
3. Are patches of IoT-centric vendors more or less likely to be available on time than those of non-IoT-centric vendors?

These questions form the bedrock of the research, designed to systematically evaluate both the preventive security performance (how many vulnerabilities exist/are reported) and reactive security performance (how well vendors respond to discovered vulnerabilities) of IoT-centric entities.

Key Findings

▶ Watch: IoT-centric vendors linked to significantly more vulnerabilities (2:47)

The research yielded several significant findings that challenge conventional wisdom and provide a nuanced view of vulnerability management in the IoT sector. The answers to the three primary research questions are particularly insightful:

1. IoT-Centric Vendors and Vulnerability Counts:

The study conclusively found that IoT-centric vendors are indeed associated with more vulnerabilities compared to non-IoT-centric vendors. The statistical model revealed a rate ratio of 6.35 for the independent variable "IoT ratio." This signifies that a one-unit increase in a vendor's IoT ratio corresponds to a 6.35-fold increase in the number of vulnerabilities published per vendor, even when controlling for other factors. This provides empirical justification for the widespread criticism aimed at IoT vendors regarding the volume of security flaws in their products.

However, the analysis also highlighted that vulnerability counts are influenced by several other factors. Larger vendors, for instance, tend to have more vulnerabilities; a tenfold increase in the number of employees was associated with a 2.41-fold increase in vulnerabilities. This is attributed not necessarily to less secure products, but to a greater number of products on the market. Furthermore, the presence of vendor policies significantly impacts reported vulnerability numbers:

• A vulnerability disclosure policy (VDP) was associated with a 3.98-fold increase in vulnerabilities.
• A bug bounty program was linked to a 5.40-fold increase.

These findings align with prior studies suggesting that vulnerability discovery is often a function of "attention effort"—meaning more effort (through policies that encourage reporting) leads to more discovered and published vulnerabilities, rather than necessarily indicating a higher actual density of vulnerabilities in the codebase.

2. Patch Availability for IoT-Centric Vendors:

Contrary to popular belief, the research found that IoT-centric vendors are neither more nor less likely to release patches for their vulnerabilities than non-IoT-centric vendors. The "IoT ratio" was not a statistically significant factor influencing the likelihood of patch availability. This suggests that while IoT-centric vendors might have more vulnerabilities, they are not inherently worse at providing fixes for those they become aware of.

Similar to vulnerability counts, other factors played a crucial role. The most significant impact on patch availability came from vulnerability disclosure policies (VDPs), which showed an odds ratio of 4.85. This means that when a vendor has a VDP, patches are 4.85 times more likely to be available compared to when they do not. While a bug bounty program was not statistically significant in this context, it generally pointed in the same positive direction, suggesting that such programs also contribute to increased patch availability.

3. Patch Timeliness for IoT-Centric Vendors:

Perhaps the most surprising finding relates to patch timeliness. The study revealed that IoT-centric vendors are more likely to publish patches in a timely manner compared to non-IoT-centric vendors. A one-unit increase in the IoT ratio correlated with a 6.56-fold increase in the likelihood of timely patch availability for vulnerabilities where a patch was published. This counter-intuitive result suggests that despite having more vulnerabilities, IoT-centric vendors demonstrate a stronger commitment or capacity for prompt remediation once a fix is developed.

However, further unexpected insights emerged regarding factors influencing timeliness:

• The presence of an exploit proof of concept (PoC) had a significant negative effect on the likelihood of timely patch availability. The researchers acknowledged the difficulty in speculating on the exact mechanisms behind this, suggesting that perhaps a PoC might demonstrate that an exploit is not trivial, leading to more complex or delayed remediation efforts.
• Vulnerability disclosure policies (VDPs) and bug bounty programs had no positive impact on patch timeliness. In fact, their direction was opposite to what one might expect, though not statistically significant. A possible explanation is that these programs primarily incentivize external researchers to discover bugs, who may then publish their findings after a set disclosure period, irrespective of whether a vendor-internal patch is ready. This suggests a disconnect between external discovery incentives and internal remediation speed.

In summary, the research concludes that while IoT-centric vendors do produce more vulnerabilities, they are not worse at releasing patches, and in fact, tend to be more timely in their patch delivery. This challenges the simplistic notion that high vulnerability counts automatically equate to poor security performance, underscoring the complex interplay of various factors in vulnerability management.

Technical Deep Dive

▶ Watch: IoT vendors not worse at releasing patches (4:50)

The rigorous methodology employed in this study is crucial for the validity of its findings. The researchers followed a four-step process to systematically analyze the security performance of IoT-centric vendors:

1. Vendor Identification and Data Collection (Vulnerabilities):

• The study began by identifying 104 important vendors in the market. The criteria for "importance" were not explicitly detailed in the transcript but imply a selection of prominent players across various sectors.
• For each vendor, the degree of IoT-centricity was quantified. This involved determining the proportion of a vendor's product portfolio that could be identified as IoT. This "IoT ratio" served as a key independent variable in the statistical models.
• Vulnerability data was collected from the National Vulnerability Database (NVD), covering 30,056 vulnerabilities published for the 104 vendors from January 2016 to November 2022.
• Crucially, vulnerabilities were classified as pertaining to IoT versus non-IoT using the BAR IoT repository. This external repository provided a standardized method for classifying CVEs, ensuring consistency and objectivity in distinguishing IoT-related security flaws.

1. Manual Data Set Construction (Patch Availability and Timeliness):

• To assess reactive security performance, the researchers manually built a unique dataset on patch availability and patch timeliness. This labor-intensive step involved analyzing a carefully selected sample of 2,741 vulnerabilities, distributed across the 104 vendors, for both IoT and non-IoT products.
• Patch availability was a binary variable, indicating whether a vendor had released a patch for a given vulnerability.
• Patch timeliness measured whether a patch was available "on time," which likely refers to a specific window after public disclosure or vendor notification, though the exact definition of "on time" was not explicitly detailed in the transcript. This manual collection ensured accuracy for these critical reactive metrics.

1. Theoretical Framework and Factor Identification:

• To explain the observed patterns in preventive and reactive security, the researchers developed a theoretical framework. This framework outlined the main factors that could influence vulnerability outcomes. These factors included intrinsic vendor characteristics and policy choices.
• Data was then collected on these identified factors. Key variables incorporated into the models included:
• IoT Ratio: The primary independent variable, quantifying a vendor's focus on IoT.
• Vendor Size: Measured by the number of employees, accounting for the scale of operations and product output.
• Vulnerability Disclosure Policy (VDP): A binary variable indicating the presence or absence of a formal VDP.
• Bug Bounty Program: A binary variable indicating the presence or absence of a bug bounty program.
• Exploit Proof of Concept (PoC) Availability: A binary variable indicating whether a public exploit PoC existed for a vulnerability.

1. Statistical Modeling:

• Statistical models were specified to estimate the impact of these various factors on the dependent variables (number of vulnerabilities, patch availability, patch timeliness).
• For the first research question (number of vulnerabilities), a model predicting counts was likely used, given the mention of a rate ratio. A rate ratio of 6.35 for the IoT ratio indicates that for every unit increase in IoT-centricity, the rate at which vulnerabilities are published increases by a factor of 6.35, holding other variables constant. This type of model is suitable for count data, often using Poisson or negative binomial regression.
• For the second and third research questions (patch availability and timeliness), which are binary outcomes (yes/no), odds ratio analysis was employed. An odds ratio of 4.85 for VDPs on patch availability means that the odds of a patch being available are 4.85 times higher when a vendor has a VDP. Similarly, an odds ratio of 6.56 for the IoT ratio on patch timeliness means the odds of a patch being timely are 6.56 times higher for more IoT-centric vendors.
• The models controlled for confounding variables (e.g., controlling for size when assessing the impact of the IoT ratio), ensuring that the observed effects were attributable to the specific factors being investigated. The use of "controlling for size and other factors" highlights the multivariate regression approach, which isolates the unique contribution of each independent variable.

This robust methodological approach, combining large-scale data collection from public databases with meticulous manual data curation and advanced statistical analysis, allowed the researchers to draw empirically sound conclusions about the complex landscape of IoT vulnerability management.

Demo / Proof of Concept

▶ Watch: IoT vendors release patches more timely than non-IoT (6:00)

This talk presented original research findings and a detailed methodological approach rather than demonstrating a live proof of concept or a specific security tool. The focus was on the systematic analysis of vendor practices through data collection and statistical modeling, rather than showcasing an exploit or a defensive mechanism in action.

Defensive Implications

▶ Watch: Summary of findings and policy implications (8:05)

The findings of this research offer crucial insights for security defenders, informing strategies for procurement, risk assessment, and policy advocacy related to IoT devices.

1. Re-evaluate Vulnerability Counts: Defenders should avoid relying solely on raw vulnerability counts as a primary metric for assessing a vendor's security posture. While IoT-centric vendors are associated with more reported vulnerabilities (a 6.35-fold increase for a one-unit increase in IoT ratio), this doesn't necessarily mean their products are inherently less secure or that they are worse at patching. A higher count can signify robust vulnerability discovery efforts, especially when supported by VDPs (3.98-fold increase) and bug bounty programs (5.40-fold increase). Defenders should instead look for transparency and active engagement in vulnerability reporting.

1. Prioritize Vendors with VDPs and Bug Bounties: The presence of a Vulnerability Disclosure Policy (VDP) significantly increases the likelihood of patches being available (4.85-fold increase in odds). While these policies don't guarantee timely patches, they are a strong indicator of a vendor's commitment to acknowledging and addressing security flaws. When procuring IoT devices, organizations should prioritize vendors that openly publish VDPs and ideally run bug bounty programs, as these foster external security research and improve overall transparency.

1. Recognize IoT-Centric Vendors' Patch Timeliness: Surprisingly, IoT-centric vendors tend to be more timely in releasing patches (a 6.56-fold increase in the likelihood of timely patch availability). This positive finding suggests that defenders should not automatically assume slower remediation from IoT specialists. This could indicate a more streamlined internal process or a greater awareness of the critical nature of IoT security for their core business. This information can influence vendor selection and ongoing trust assessments.

1. Understand the Nuances of Exploit PoCs: The counter-intuitive finding that the presence of an exploit proof of concept (PoC) negatively impacts patch timeliness is important. While defenders should always prioritize patching vulnerabilities with public exploits due to immediate risk, they should be aware that such vulnerabilities might take longer for vendors to fix. This could be due to the complexity of the exploit, requiring more extensive redesign or testing, rather than a lack of intent. Defenders should factor this potential delay into their incident response planning and consider compensatory controls.

1. Advocate for Evidence-Based Policy: The study provides an "evidence-based rational and legitimacy for developing interventions aimed at IoT-centric vendors." Defenders, especially those involved in industry groups or regulatory bodies, can use these findings to advocate for targeted policies that encourage robust vulnerability management practices, focusing on areas where IoT vendors genuinely lag (e.g., initial vulnerability density) while acknowledging their strengths (e.g., patch timeliness).

1. Focus on Continuous Monitoring and Risk Assessment: Given the complex dynamics, defenders must implement continuous monitoring strategies for their IoT deployments. This includes tracking vendor patch releases, staying informed about public vulnerability disclosures, and performing regular risk assessments tailored to the specific IoT devices in use, rather than relying on generalized assumptions about "IoT security."

Key Takeaways

• IoT-centric vendors are associated with significantly more vulnerabilities: The study found a 6.35-fold increase in vulnerabilities for a one-unit increase in IoT ratio, providing empirical evidence for common concerns about IoT security.
• Vulnerability Disclosure Policies (VDPs) and bug bounty programs boost reported vulnerabilities and patch availability: These programs lead to a 3.98-fold increase in vulnerabilities (VDP) and a 4.85-fold increase in patch availability (VDP), indicating that increased "attention effort" drives discovery and initial remediation.
• IoT-centric vendors are NOT worse at releasing patches and are MORE timely: Despite higher vulnerability counts, IoT-centric vendors are just as likely to release patches and are 6.56 times more likely to release them on time compared to non-IoT-centric vendors.
• Vulnerability counts alone are insufficient for assessing vendor security: High vulnerability numbers can reflect strong discovery efforts rather than inherently poorer security, highlighting the need for a holistic evaluation of vendor practices.
• Exploit PoCs can negatively impact patch timeliness: Counter-intuitively, the presence of an exploit proof of concept was associated with a decrease in patch timeliness, potentially due to the complexity of the demonstrated vulnerability.
• Evidence-based policy is crucial for IoT security: The findings challenge assumptions and provide a solid empirical basis for developing targeted interventions and policies to improve vulnerability management for IoT devices.

About the Speaker(s)

Sandra Rivera Pérez is a researcher from Delft University of Technology, where she presented this work. Her research focuses on the vulnerability management practices of IoT-centric vendors, supported by the Intersect project.

Michel van Eeten is a collaborator on this research, presumably a professor or senior researcher at Delft University of Technology, given the academic context. His involvement suggests expertise in cybersecurity policy, governance, or empirical security studies.

Carlos H. Gañán is also a collaborator on this study, likely a researcher or professor at Delft University of Technology. His contribution would typically involve technical aspects of the research, methodology, or data analysis.

The team's affiliation with Delft University of Technology and their involvement in the Intersect project underscore their background in academic research focused on cybersecurity and critical infrastructure.