Advances and Open Problems in Federated Learning

This paper, “Advances and Open Problems in Federated Learning” (Foundations and Trends® in Machine Learning, 2020), is a broad survey and research roadmap rather than a single empirical study. Its central “research question” is: what are the defining characteristics of federated learning (FL), what progress has been made across the main subareas, and—most importantly—what open problems remain that are likely to matter for both theory and real-world deployments? This matters because FL is now used in production settings (e.g., mobile keyboard and messaging features), yet the setting differs sharply from standard centralized training: data are decentralized, clients are unreliable and intermittently available, communication is constrained, and privacy and security requirements are first-order.

The paper frames FL as a machine learning setting where many clients collaborate under orchestration of a central server while keeping raw data local. A key contribution is the paper’s operational definition emphasizing “focused updates” and early aggregation: clients send only narrowly scoped updates needed for the learning objective, and aggregation happens as early as possible to support data minimization. It distinguishes cross-device FL (massively many, unreliable mobile/IoT clients) from cross-silo FL (a small number of organizations, more reliable, often with different incentives and constraints). It also situates FL among related paradigms such as fully decentralized peer-to-peer learning and split learning.

Methodologically, the paper synthesizes prior work and organizes it into a taxonomy of challenges and solution directions. It uses theoretical convergence results and privacy/security formalisms as anchors, but it does not introduce new experiments or a new dataset. Instead, it provides: (i) a canonical FL training process template (client selection, broadcast, local computation, aggregation, server update), (ii) a taxonomy of non-IID data regimes, (iii) a synthesis of optimization convergence-rate results for IID vs non-IID settings, (iv) a structured overview of privacy technologies (MPC, homomorphic encryption, TEEs, secure aggregation/shuffling, differential privacy variants), (v) an attack/failure-mode landscape (poisoning, backdoors, evasion, Byzantine failures, system failures), (vi) fairness and bias considerations (including fairness without sensitive attributes), and (vii) system-level challenges (deployment, monitoring, bias induced by device availability, tuning, and on-device runtime).

Across these sections, the paper highlights several concrete quantitative findings from the literature it surveys. For example, in the IID convex setting under bounded stochastic gradient variance, it summarizes convergence-rate upper bounds for federated averaging/local SGD and related baselines, emphasizing that federated methods can match the “statistical” term but often have a suboptimal “optimization” term relative to accelerated minibatch SGD. It also provides scaling heuristics: in typical cross-device regimes, the number of clients per round can be on the order of hundreds while the number of rounds can be very large, implying that the number of local steps per round often must be small to preserve convergence guarantees. In the non-IID setting, it reports that achieving the IID-like rate requires stricter constraints on the number of local updates: where IID analysis allows local steps up to roughly a certain order, non-IID analysis typically requires smaller local-update budgets (the paper states that to achieve an error bound scaling like $1/\sqrt{TKN}$ under non-convex objectives, the number of local updates $K$ should be smaller than $T^{1/3}/N$ rather than the IID threshold $T/N^3$). It also provides a table of representative non-IID convergence rates for methods such as FedProx, SCAFFOLD, and others, along with the assumptions used (e.g., bounded inter-client gradient variance, bounded optimal objective difference, bounded gradient dissimilarity).

In privacy, the paper is explicit about formal definitions and mechanisms. It introduces differential privacy (DP) with parameters $(\epsilon ,\delta )$ and gives the DP inequality, then maps FL threat models to “what” is computed (privacy-preserving disclosure) and “how” it is computed (secure computation and verifiability). It distinguishes user-level DP (adjacency by adding/removing all records of a user) from record-level DP, and it explains why FL often needs user-level guarantees. It also discusses privacy accounting challenges in cross-device FL with dynamic eligibility and dropout, and it highlights that distributed DP mechanisms often require discrete noise distributions that complicate standard DP accounting tools like the moments accountant.

In security/robustness, the paper’s quantitative specificity is more limited because it is a survey, but it still cites concrete numeric examples from the literature. For instance, it notes that targeted backdoors can be introduced even with anomaly detectors, and it cites a result that if 10% of devices participating in federated learning are compromised, a backdoor can be introduced by poisoning model updates sent to the service provider (attributed to Bhagoji et al.). It also characterizes adversaries along multiple axes (attack vector, model inspection capability, collusion, participation rate, adaptability), which is crucial for interpreting which defenses apply.

Limitations are inherent to the paper’s survey nature: it does not provide a unified new algorithm, dataset, or experimental evaluation. Instead, it acknowledges that FL research is difficult to validate because many real-world constraints (client availability, communication topology, privacy-preserving evaluation, and non-IID structure) are hard to reproduce. It also emphasizes that simulations must be carefully justified: success on a proxy dataset does not automatically imply success in the real deployment objective. Additionally, because the paper is interdisciplinary, it necessarily leaves some topics less quantified than others (e.g., fairness metrics and system-induced bias quantification are discussed as open problems rather than resolved).

Practical implications are extensive. For ML researchers, the paper provides a checklist for specifying the FL setting precisely (cross-device vs cross-silo; client statefulness; availability patterns; non-IID regime), for reporting where computation happens and what is communicated, and for using privacy and communication efficiency as first-order evaluation criteria even in simulations. For system builders, it highlights that deployment, monitoring, and debugging are major bottlenecks, and that device availability constraints can induce bias. For policy and product stakeholders, it clarifies that FL improves privacy primarily through data minimization but does not automatically guarantee privacy without formal mechanisms (DP, secure aggregation, etc.), and that privacy guarantees depend on threat models.

Overall, the paper’s core message is that federated learning is not a single algorithmic trick but a full stack of interacting constraints—optimization, communication, privacy, security, fairness, and systems—that create new failure modes and new research frontiers. The most important finding, in the sense of the paper’s contribution, is the structured mapping from FL’s defining constraints to the open problems that remain unsolved across these layers, along with concrete pointers to the theoretical and practical tools that are most promising to combine next.