PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
PFAdapter achieves communication-efficient and semantically decoupled personalization of Multimodal Large Language Models (MLLMs) in federated learning by explicitly decomposing LoRA adapter parameters into global-shared and local-private components, significantly reducing communication overhead while improving edge-specific adaptation performance.
Core Contribution
PFAdapter introduces a hierarchical Low-Rank Adaptation (LoRA) decomposition mechanism—the first such design for federated Multimodal Large Language Models (MLLMs)—that jointly optimizes global semantic consistency and local personalization. It abandons conventional uniform parameter aggregation in favor of a function-aware, semantic-driven separation of LoRA parameters within attention modules.
Technical Approach
- LoRA weights for Query and Key projections are designated as global-shared components, synchronized server-side to capture universal multimodal semantics across clients (e.g., cross-modal alignment patterns);
- LoRA weights for Value and Output projections remain local-private components, updated exclusively on edge devices to adapt to client-specific data distributions (e.g., camera viewpoints, sensor noise profiles, or user interaction patterns);
- Only Query/Key-related low-rank matrices are transmitted; theoretical communication cost is reduced by ~63% versus full-parameter FedAvg (per arXiv:2607.12111v1 experimental setup).
Problem Context
Existing federated protocols (e.g., FedAvg, FedProx) face dual bottlenecks in MLLM settings: (1) uniform aggregation conflates domain-invariant features with client-specific nuances, degrading personalization; (2) frequent upload/download of high-dimensional multimodal model parameters exacerbates bandwidth pressure on edge networks. PFAdapter directly addresses this trade-off via modular parameter decomposition—not coarse-grained weight averaging.