Stein Variational Gradient Descent and Consensus-Based Optimization: Towards a Convergence Analysis and Generalization Lukang Sun, Ph.D. Student, Computer Science May 30, 11:00 - 14:00 B3 L5 R5220 The first part of the dissertation presents a study on the convergence properties of Stein Variational Gradient Descent (SVGD), a sampling algorithm with applications in machine learning. The research delves into the theoretical analysis of SVGD in the population limit, focusing on its behavior under various conditions, including the Talagrand’s inequality T1 and the (L0, L1)−smoothness condition. The study also introduces an improved version of SVGD with importance weights, demonstrating its potential to accelerate convergence and enhance stability.
Better Methods and Theory for Federated Learning: Compression, Client Selection, and Heterogeneity Samuel Horváth, Ph.D. Student, Statistics Jun 27, 18:00 - 20:00 B5 L5 R5209 Federated learning Optimization for Machine Learning Federated learning (FL) is an emerging machine learning paradigm involving multiple clients, e.g., mobile phone devices, with an incentive to collaborate in solving a machine learning problem coordinated by a central server. FL was proposed in 2016 by Konecny et al. and McMahan et al. as a viable privacy-preserving alternative to traditional centralized machine learning since, by construction, the training data points are decentralized and never transferred by the clients to a central server. Therefore, to a certain degree, FL mitigates the privacy risks associated with centralized data collection. Unfortunately, optimization for FL faces several specific issues that centralized optimization usually does not need to handle. In this thesis, we identify several of these challenges and propose new methods and algorithms to address them, with the ultimate goal of enabling practical FL solutions supported with mathematically rigorous guarantees.
Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters Filip Hanzely, Ph.D., Applied Mathematics and Computational Sciences Jul 30, 16:00 - 18:00 KAUST Many key problems in machine learning and data science are routinely modeled as optimization problems and solved via optimization algorithms. With the increase of the volume of data and the size and complexity of the statistical models used to formulate these often ill-conditioned optimization tasks, there is a need for new efficient algorithms able to cope with these challenges. In this thesis, we deal with each of these sources of difficulty in a different way. To efficiently address the big data issue, we develop new methods which in each iteration examine a small random subset of the training data only. To handle the big model issue, we develop methods which in each iteration update a random subset of the model parameters only. Finally, to deal with ill-conditioned problems, we devise methods that incorporate either higher-order information or Nesterov's acceleration/momentum. In all cases, randomness is viewed as a powerful algorithmic tool that we tune, both in theory and in experiments, to achieve the best results.
Proximal Splitting Methods for Convex Optimization: An Introduction Laurent Condat, Senior Research Scientist, Computer, Electrical and Mathematical Sciences and Engineering Dec 2, 12:00 - 13:00 B9 L2 H1 R2322 This talk will be a gentle introduction to proximal splitting algorithms to minimize a sum of possibly nonsmooth convex functions. Several such algorithms date back to the 60s, but the last 10 years have seen the development of new primal-dual splitting algorithms, motivated by the need to solve large-scale problems in signal and image processing, machine learning, and more generally data science. No background will be necessary to attend the talk, whose goal is to present the intuitions behind this class of methods.
Laurent Condat, Senior Research Scientist, Computer, Electrical and Mathematical Sciences and Engineering
Langevin Monte Carlo as an optimization algorithm Adil Salim, Postdoctoral Research Fellow, Computer Science Nov 11, 12:00 - 13:00 B9 L2 H1 R2322 machine learning Langevin Monte Carlo optimization Adil Salim is mainly interested in stochastic approximation, optimization, and machine learning. He is currently a Postdoctoral Research Fellow working with Professor Peter Richtarik at the Visual Computing Center (VCC) at King Abdullah University of Science and Technology (KAUST).
