DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training

DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training

Speakers: Zihao Wang, Yale School of Medicine; Zhikun Zhang, Indiana University; John Mitchell, Stanford University; Haixu Tang, Indiana University; XiaoFeng Wang, Indiana University

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=Qxjfso2UfNg

Overview

In the realm of deep learning, where models are increasingly deployed to process sensitive information, ensuring data privacy has become paramount. This talk introduces DPAdapter, a novel methodology designed to significantly enhance the utility of Differentially Private Machine Learning (DPM), specifically in the context of Differentially Private Stochastic Gradient Descent (DPSGD). The core innovation of DPAdapter lies in optimizing the pre-training phase of deep learning models, making them inherently more robust to the noise injection required for differential privacy.

Presented by Zihao Wang and a collaborative team from Yale School of Medicine, Indiana University, and Stanford University, DPAdapter addresses a critical challenge in privacy-preserving AI: the trade-off between privacy guarantees and model performance. By focusing on creating a "flatter loss landscape" during pre-training, DPAdapter enables downstream users to fine-tune models on private datasets with substantially improved accuracy, all while maintaining strict differential privacy guarantees. This work is particularly relevant for Machine Learning as a Service (MLaaS) scenarios, where upstream model vendors can leverage DPAdapter to provide privacy-ready foundation models, empowering downstream users to apply DPM more effectively.

Background

▶ Watch: Motivation: Mitigating DP-SGD challenges (2:40)

Deep learning models, while powerful, are notoriously "data-hungry" and prone to memorizing their training samples. This characteristic poses a significant privacy risk: without robust protections, an attacker could potentially extract sensitive information from the model's parameters. To counter this, Differential Privacy (DP) has emerged as a gold standard, offering strong, mathematically provable guarantees against such data extraction.

Among the most prominent approaches to integrating DP with deep learning is Differentially Private Stochastic Gradient Descent (DPSGD). The mechanism of DPSGD is conceptually straightforward:

1. During model training, a batch of data is selected.
2. The model's gradient, representing the direction and magnitude of parameter updates, is calculated based on this batch and the loss function.
3. A crucial step is gradient clipping, where large gradient components are truncated to ensure each component remains bounded. This prevents individual data points from disproportionately influencing the model update.
4. Finally, Laplace noise is added to the clipped gradient. This noise is calibrated to satisfy the differential privacy guarantee, ensuring that any single data point in the batch cannot be inferred from the resulting noisy gradient.

Despite its strong privacy guarantees, DPSGD comes with considerable drawbacks. The noise injection, while necessary for privacy, often leads to a significant reduction in model utility, making the model perform less effectively on its intended tasks. Furthermore, the noise can destabilize the training process, making it harder for the model to converge smoothly and requiring extensive, careful hyperparameter tuning to achieve acceptable results. These challenges make the practical deployment of DPM difficult and resource-intensive.

The motivation behind DPAdapter stems from the need to mitigate these issues. The researchers' core question was: Given relevant public datasets, can we identify a pre-training starting point in a flatter loss landscape such that the subsequent impact of adding DP noise on the gradients of private data does not significantly shift the loss? This question is particularly pertinent in the modern MLaaS supply chain, where upstream model vendors train foundational models on large public datasets, and downstream users then fine-tune these models using their own sensitive, private data. DPAdapter specifically targets the upstream pre-training process, aiming to optimize it to enhance the effectiveness of DPM when deployed by downstream users, ultimately striving for better utility with the same privacy budget.

Key Findings

▶ Watch: Detailed explanation of DPAdapter's four steps (4:40)

DPAdapter demonstrates compelling empirical and theoretical evidence for its effectiveness in improving differentially private deep learning. The key findings highlight its ability to boost model utility, its broad applicability, and its complementary nature to existing DPM techniques.

Empirically, DPAdapter consistently outperformed models pre-trained with vanilla methods across various experimental setups. The observed improvements in accuracy ranged from a significant 3% to 16%, showcasing its tangible impact on model utility. This performance boost was validated across:

• Four different DPM methods (excluding vanilla DPSGD, but enhancing its application).
• Four different pre-training methods (including training from scratch, normal pre-training, parameter robustness training with random perturbation, and DPAdapter).
• Three distinct downstream tasks evaluated on the CIFAR-100 dataset.

The research further evaluated the stability and performance of DPAdapter across different privacy budgets, public datasets, and model architectures. Using upstream training datasets like CIFAR-100 and Tiny ImageNet, and leveraging sophisticated model architectures such as the Vision Transformer (ViT) and the robust ResNet framework, DPAdapter consistently demonstrated its ability to enhance downstream DML outcomes, particularly on the STL-10 dataset. This broad validation underscores DPAdapter's generalizability and robustness.

A particularly significant finding is DPAdapter's potential for combination with other existing DPM enhancement techniques. Since DPAdapter optimizes the model from a unique angle—the upstream pre-training phase, rather than the DPM phase itself—it naturally complements downstream optimizations. When DPAdapter was used to enhance a pre-trained model and then paired with an existing DPM enhancement technique called MGC (Momentum Gradient Clipping) during downstream training, the combination yielded an impressive 12% improvement in accuracy compared to using MGC with a standard pre-trained model alone. This demonstrates that DPAdapter can serve as an independent, foundational layer of improvement.

The researchers also provided a theoretical understanding for DPAdapter's effectiveness, articulating two main theorems:

• Theorem 1: This theorem establishes that under certain reasonable constraints, the batch size B2 (used when computing the perturbation during DPAdapter's training) is directly proportional to the upper bound of the model's convergence rate. This implies that using a larger B2 batch size is likely to lead to a better optimization minimum, contributing to improved model performance.
• Theorem 2: This theorem provides a stochastic explanation for the effect of DPAdapter. It posits that starting the DPSGD process from a point with higher parameter robustness (i.e., a flatter region in the loss landscape, as achieved by DPAdapter) enables DPSGD to converge to a minimum with a lower loss. This theoretical insight validates the core hypothesis that an optimized pre-training landscape is beneficial for DPM.

These findings collectively highlight DPAdapter as a powerful and versatile approach to overcome the utility degradation associated with differential privacy, paving the way for more practical and effective privacy-preserving machine learning deployments.

Technical Deep Dive

▶ Watch: DPAdapter's unique pre-DPM phase focus (6:00)

DPAdapter's methodology is structured around a four-step iterative process, drawing inspiration from sharpness-aware minimization (SAM) techniques. The overarching goal is to improve the parameter robustness of the pre-trained model, effectively sculpting a "flatter loss landscape." This is crucial because DPM methods, particularly DPSGD, intentionally inject noise into the model's gradients. If the model's parameters are more resistant to such noise—meaning small perturbations in parameters do not lead to large changes in loss—then the impact of the added DP noise will be less detrimental to utility.

Let's break down the four steps:

1. Step 1: Warm-up Training:

The process begins with a standard warm-up phase. The model is trained for a few epochs using the available public training data. This initial phase helps the model to converge to a reasonable starting point in the loss landscape before the robustness optimization begins.

1. Step 2: Calculate Worst-Case Perturbation:

This is a critical step for enhancing parameter robustness. The method selects a batch of B1 samples from the public data. The objective here is to find a gradient update (a perturbation) that maximizes the increase in loss for this specific batch. Conceptually, this is the opposite direction of a normal gradient update, which aims to minimize loss. By identifying and applying this "worst-case" perturbation, the model is pushed towards a region where it is more sensitive to changes. This perturbation is then added to the current model parameters. The researchers emphasize that B1 should be relatively large, a point later justified by their theoretical findings (Theorem 1). This large batch size helps in accurately estimating the direction of maximum loss increase.

1. Step 3: Optimize the Perturbed Model:

After applying the worst-case perturbation, the model parameters are now in a "perturbed" state. The next step involves selecting a new batch of B2 samples. The perturbed model is then optimized using this new batch. This optimization step aims to find a robust minimum from the perturbed state, effectively training the model to recover and perform well even when its parameters have been intentionally shifted.

1. Step 4: Remove Perturbation and Iterate:

Similar to the fashion of most sharpness-aware optimization techniques, once the optimization in Step 3 is complete, the worst-case perturbation that was added in Step 2 is removed from the model parameters. The process then repeats, cycling through Steps 2 to 4 until the model converges. This iterative application of perturbing, optimizing, and restoring parameters iteratively pushes the model towards a flatter, more robust region of the loss landscape.

A key distinction of DPAdapter is its focus on the pre-DPM phase. Unlike many traditional DPM enhancement techniques that primarily optimize during the DPM training process itself (e.g., by refining gradient clipping strategies or noise mechanisms), DPAdapter optimizes the upstream model. This means the resulting pre-trained model is inherently more resilient to noise before any differential privacy mechanisms are even applied. This distinction makes DPAdapter highly adaptable and easy to combine with existing downstream DPM enhancement techniques, as demonstrated by its successful integration with MGC. The theoretical underpinnings, particularly Theorem 2, reinforce that starting from a state of higher parameter robustness (a flatter loss landscape) directly leads to a lower loss minimum when DPSGD is subsequently applied.

Demo / Proof of Concept

▶ Watch: Enhanced performance by combining with other DPM techniques (8:00)

While the talk did not feature a live, interactive demonstration of the DPAdapter system in action, the speakers presented extensive empirical evidence and comparative analyses to validate its effectiveness as a proof of concept. The core of their "demonstration" involved rigorous experimental evaluation across a variety of settings to showcase DPAdapter's performance improvements.

The evaluation framework was comprehensive:

• DPM Methods: DPAdapter's efficacy was tested in conjunction with four different DPM methods, demonstrating its broad applicability beyond a single specific differentially private algorithm.
• Pre-training Baselines: The performance of models pre-trained with DPAdapter was compared against several baselines: training from scratch, standard (vanilla) pre-training, and pre-training enhanced with random perturbation (a simpler form of robustness training).
• Datasets and Tasks: Experiments were conducted on the well-known CIFAR-100 dataset for upstream training and evaluated on three different downstream tasks. This allowed for assessing DPAdapter's ability to transfer robustness to various fine-tuning scenarios.
• Quantified Improvements: The results showed that DPAdapter consistently outperformed models pre-trained with the vanilla method, yielding accuracy improvements ranging from 3% to 16%. This direct quantification of utility gain serves as a strong proof of concept.

Further experiments explored the impact of different public upstream training datasets and model architectures on DPAdapter's stability and performance.

• Upstream Data: Comparisons were made using CIFAR-100 and Tiny ImageNet as upstream training datasets.
• Model Architectures: The evaluation included both sophisticated Transformer architectures (ViT) and robust Convolutional Neural Network (CNN) frameworks (ResNet).
• Downstream Evaluation: The accuracy was measured in an announcement task on a fixed STL-10 dataset. Across all these variations, DPAdapter consistently demonstrated its ability to enhance downstream DML outcomes, reinforcing its robustness and generalizability.

Perhaps the most compelling aspect of the proof of concept was the demonstration of DPAdapter's synergistic effect when combined with other DPM enhancement techniques. By improving the pre-trained model with DPAdapter and then coupling it with MGC (Momentum Gradient Clipping) during downstream DPM, the researchers observed a significant 12% improvement in accuracy over using MGC with a standard pre-trained model alone. This result powerfully illustrates DPAdapter's role as a foundational, complementary improvement that can be layered with existing methods to achieve even greater utility. These extensive empirical results collectively serve as a robust proof of concept for DPAdapter's efficacy in enhancing differentially private deep learning.

Defensive Implications

▶ Watch: Theoretical understanding: Theorems for DPAdapter effectiveness (8:50)

DPAdapter offers significant defensive implications for organizations and practitioners involved in developing and deploying machine learning models, particularly those handling sensitive data. Its core contribution shifts the focus of DPM enhancement to the pre-training phase, providing a new avenue for strengthening privacy guarantees without severely compromising model utility.

For upstream model vendors in the MLaaS supply chain, DPAdapter presents a clear strategy. By integrating DPAdapter into their pre-training pipelines, vendors can produce foundation models that are inherently more robust to noise. These "privacy-ready" models would then serve as superior starting points for downstream users who need to fine-tune them with private data. This means vendors can offer a competitive advantage by providing models that enable downstream users to achieve higher accuracy for a given privacy budget (epsilon) or maintain a desired accuracy level with a tighter (more stringent) privacy budget. This proactive approach by vendors alleviates some of the burden on downstream users.

For downstream users who are tasked with fine-tuning models on their sensitive, private datasets, DPAdapter-enhanced pre-trained models offer a substantial benefit. Instead of struggling with the inherent utility degradation and convergence issues of vanilla DPSGD, they can leverage a foundation model that is already optimized for noise tolerance. This translates to:

• Higher utility: Achieving better accuracy on their specific tasks with the same level of differential privacy.
• Easier training: Potentially smoother convergence and reduced need for extensive hyperparameter tuning, making the DPM adoption process less arduous.
• Enhanced privacy-utility trade-off: More flexibility in balancing privacy requirements with performance goals.

Furthermore, the finding that DPAdapter is complementary to existing DPM enhancement techniques (like MGC) means that defenders should not view it as a replacement, but rather as an additional layer of defense. By combining DPAdapter at the pre-training stage with other state-of-the-art DPM techniques during fine-tuning, organizations can unlock even greater performance gains for their privacy-preserving models. This suggests a multi-faceted defensive strategy where robustness is built in from the ground up.

In essence, DPAdapter highlights the critical importance of parameter robustness and the loss landscape in the context of DPM. Defenders should recognize that optimizing the initial state of a model—making it less sensitive to the perturbations caused by privacy noise—is a powerful and underutilized strategy. This paradigm shift encourages a holistic view of privacy-preserving machine learning, where privacy considerations are integrated not just at the final training step, but throughout the entire model lifecycle, starting from its very inception during pre-training.

Key Takeaways

• Robust Pre-trained Models are Key for DPM: Pre-trained models with better parameter robustness (i.e., a flatter loss landscape) significantly enhance the utility of downstream Differentially Private Machine Learning (DPM) tasks. A solid, noise-tolerant foundation improves everything built upon it.
• Batch Size Matters for Robustness Optimization: The batch sizes (B1 and B2) used during the perturbation and optimization steps of DPAdapter are crucial. A larger batch size (B2) in particular contributes to better optimization and convergence rates, leading to a more robust model.
• Parameter Robustness as an Independent Resource: The parameter robustness of a pre-trained model serves as an independent and beneficial resource for DPM. It offers advantages beyond what existing DPM methods typically focus on (which often center on optimizing the downstream task directly).
• DPAdapter Improves ML-as-a-Service: DPAdapter provides a practical and effective method for upstream model vendors to optimize their pre-training process, enabling them to deliver more robust foundational models that empower downstream users to achieve higher utility with the same privacy budget when fine-tuning on sensitive data.
• Complementary to Existing DPM Techniques: DPAdapter's optimization in the pre-DPM phase makes it highly complementary to existing DPM enhancement techniques. Combining DPAdapter with other methods (e.g., MGC) can yield even greater improvements in accuracy and privacy-utility trade-offs.
• Holistic Approach to Privacy: The research underscores that focusing on the pre-training phase is a powerful and often overlooked strategy for improving DPM, advocating for a holistic approach where privacy considerations are integrated from the very beginning of the model development lifecycle.

About the Speaker(s)

The talk was presented by Zihao Wang, affiliated with the Yale School of Medicine. Zihao Wang is part of a collaborative research effort that includes colleagues from several esteemed institutions.

The co-authors of this work include:

• Zhikun Zhang from Indiana University
• John Mitchell from Stanford University
• Haixu Tang from Indiana University
• XiaoFeng Wang from Indiana University

This collaborative team brings together expertise from diverse academic backgrounds, including medicine, computer science, and security research, indicating a multidisciplinary approach to addressing complex challenges in deep learning privacy. Their collective work focuses on advancing the state-of-the-art in differentially private machine learning, particularly in developing robust and practical solutions that can be deployed in real-world scenarios. Their research contributes significantly to enhancing the security and privacy of AI systems, a critical area given the increasing deployment of machine learning models in sensitive domains.