SILO: Csaba Szepesvari
2022-01-26 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

IFDS Ideas Forum
2022-01-31 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

IFDS Ideas Forum
2022-01-31 14:00 (America/Chicago)
Lightning talks for ICML+COLT submissions

SILO: Kassem Fawaz
2022-02-02 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

IFDS Monthly All-Hands: Rebecca Willett
2022-02-04 12:30 - 13:30 (America/Los_Angeles)
Speaker: Prof. Rebecca Willett, Statistics and CS, University of Chicago
Title: The Role of Linear Layers in Nonlinear Interpolating Networks
Abstract: In this discussion, we will explore the implicit bias of overparameterized neural networks of depth greater than two layers. Our framework considers a family of networks of varying depth that all have the same capacity but different implicitly defined representation costs. The representation cost of a function induced by a neural network architecture is the minimum sum of squared weights needed for the network to represent the function; it reflects the function space bias associated with the architecture. Our results show that adding linear layers to a ReLU network yields a representation cost that reflects a complex interplay between the alignment and sparsity of ReLU units. Specifically, using a neural network to fit training data with minimum representation cost yields an interpolating function that is constant in directions perpendicular to a low-dimensional subspace on which a parsimonious interpolant exists. This is joint work with Greg Ongie.

IFDS Ideas Forum: Rare Gems: Finding Lottery Tickets at Initialization
2022-02-07 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States
Title: Rare Gems: Finding Lottery Tickets at InitializationSpeaker: Jy-Yong SoonTime+location: Monday 7 Feb 2022, 12:30-13:30 Orchard View Room

Abstract: It has been widely observed that large neural networks can be pruned to a small fraction of their original size, with little loss in accuracy, by typically following a time-consuming “train, prune, re-train” approach. Frankle and Carbin in 2019 conjecture that we can avoid this by training lottery tickets, i.e., special sparse subnetworks found at initialization, that can be trained to high accuracy. However, a subsequent line of work presents concrete evidence that current algorithms for finding trainable networks at initialization, fail simple baseline comparisons, e.g., against training random sparse subnetworks. Finding lottery tickets that train to better accuracy compared to simple baselines remains open. In this work, we resolve this open problem by discovering rare gems: subnetworks at initialization that attain considerable accuracy, even before training. Refining these rare gems – by means of fine-tuning – beats current baselines and leads to accuracy competitive or better than magnitude pruning methods.Bio:Jy-yong is a post-doctoral researcher in the Department of Electrical and Computer Engineering (ECE) at the University of Wisconsin-Madison, working with Prof. Dimitris Papailiopoulos and Prof. Kangwook Lee. He is interested in the intersection of machine learning, information theory, and distributed algorithms. He received his Ph.D. degree in 2020 from KAIST, under the supervision of Prof. Jaekyun Moon. He is a recipient of the IEEE ICC Best Paper Award, Qualcomm Innovation Awards, and NRF Korea Post-doctoral Fellowship.

IFDS Ideas Forum
2022-02-07 14:00 (America/Chicago)
Rare Gems: Finding Lottery Tickets at Initialization

SILO: Yea-Seul Kim
2022-02-09 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

ML Opt@ UW: Vincent Roulet
2022-02-11 12:30 - 13:30 (America/Los_Angeles)
Speaker: Vincent Roulet
Title: Complexity Bounds of Iterative Linearization Algorithms for Discrete-Time Nonlinear Control
Abstract: We revisit the nonlinear optimization approach to discrete-time nonlinear control and optimization algorithms based on iterative linearization. While widely popular in many domains, these algorithms have mainly been analyzed from an asymptotic viewpoint. We establish non-asymptotic complexity bounds and global convergence for a class of generalized Gauss-Newton algorithms relying on iterative linearization of the nonlinear control problem, henceforth calling iterative linear quadratic regulator or differential dynamic programming algorithms as subroutines. The sufficient conditions for global convergence are examined for multi-rate sampling schemes given the existence of a feedback linearization scheme. We illustrate the algorithms in synthetic experiments and provide a software library based on reverse-mode automatic differentiation to reproduce the numerical results.

IFDS Ideas Forum: TBD
2022-02-14 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

IFDS Ideas Forum
2022-02-14 14:00, 2022-02-21 14:00, 2022-03-14 14:00 (America/Chicago)

E & A SIG:
2022-02-16 12:00 - 13:00 (America/Los_Angeles)

SILO: Victor M Zavala
2022-02-16 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

ML Opt @ UW: Yifang Chen
2022-02-18 12:30 - 13:30 (America/Los_Angeles)
Speaker: Yifang Chen
Title: Active Multi-Task Representation Learning
Abstract: To leverage the power of big data from source tasks and overcome the scarcity of the target task samples, representation learning based on multi-task pretraining has become a standard approach in many applications. However, up until now, choosing which source tasks to include in the multi-task learning has been more art than science. In this paper, we give the first formal study on resource task sampling by leveraging the techniques from active learning. We propose an algorithm that iteratively estimates the relevance of each source task to the target task and samples from each source task based on the estimated relevance. Theoretically, we show that for the linear representation class, to achieve the same error rate, our algorithm can save up to a textit{number of source tasks} factor in the source task sample complexity, compared with the naive uniform sampling from all source tasks. We also provide experiments on real-world computer vision datasets to illustrate the effectiveness of our proposed method on both linear and convolutional neural network representation classes.

IFDS Ideas Forum: Ankit Pensia & Thanasis Pittas
2022-02-21 12:30 - 13:30 (America/Chicago) | Zoom
THIS IS A ZOOM ONLY EVENT. NO LIVE AUDIENCE.
Title: Hypothesis Testing under Communication Constraints
Speaker: Ankit Pensia
Abstract: Simple hypothesis testing is a fundamental problem in statistics and it is well-known that its sample complexity is characterized by the Hellinger distance between the two candidate distributions. In this talk, we discuss the problem of simple hypothesis testing under communication constraints, wherein each sample is mapped to a message from a finite set of messages before being revealed to the statistician. We show that it is possible to map samples to messages such that the sample complexity is only an extra logarithmic factor larger than the non-constrained setting. Our proofs rely on a reverse data processing inequality and a reverse Markov’s inequality, which might be of independent interest. This is joint work with Po-Ling Loh and Varun Jog.
Bio: Ankit Pensia is a graduate student in the CS department. He is interested in robust statistics, learning theory, and high-dimensional statistics. Website: https://ankitp.net

Title: Streaming Algorithms for High-Dimensional Robust Statistics
Speaker: Thanasis Pittas
Abstract: We study high-dimensional robust statistics tasks in the streaming model. A recent line of work obtained computationally efficient algorithms for a range of high-dimensional robust statistics tasks. Unfortunately, all previous algorithms require storing the entire dataset, incurring memory at least quadratic in the dimension. In this work, we develop the first efficient streaming algorithms for high-dimensional robust statistics with near-optimal memory requirements (up to logarithmic factors). Our main result is for the task of high-dimensional robust mean estimation in (a strengthening of) Huber’s contamination model. We give an efficient single-pass streaming algorithm for this task with near-optimal error guarantees and space complexity nearly-linear in the dimension. As a corollary, we obtain streaming algorithms with near-optimal space complexity for several more complex tasks, including robust covariance estimation, robust regression, and more generally robust stochastic optimization.
Bio: Thanasis Pittas is a PhD student at the University of Wisconsin-Madison, advised by Prof. Ilias Diakonikolas. He works on theoretical machine learning and robust statistics. Thanasis was an IFDS RA during the summer of 2021. He did his undergraduate studies in Greece, at the National Technical University of Athens.

SILO: Mengdi Wang
2022-02-23 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

ML-Opt @ UWash: Krishna Pillutla
2022-02-25 12:30 - 13:30 (America/Chicago)
Title: MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
Abstract: As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that MAUVE identifies known properties of generated text, scales naturally with model size, and correlates with human judgments, with fewer restrictions than existing distributional evaluation metrics.

IFDS Ideas Forum: RA Talk
2022-02-28 00:00 (America/Chicago) | Zoom
Jiaxin Hu & Yuchen Zeng

IFDS Ideas Forum
2022-02-28 14:00 (America/Chicago)
A Review of Iterate Retraction in First Order Optimization

SILO: Mario Figueiredo
2022-03-02 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

IFDS All-Hands: Zaid Harchaoui
2022-03-04 12:30 - 13:30 (America/Los_Angeles)

IFDS Ideas Forum: Multiscale inverse problem, from Schroedinger to Newton to Boltzmann
2022-03-07 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States
Title: Multiscale inverse problem, from Schroedinger to Newton to Boltzmann
Speaker: Qin Li, Department of Mathematics
Date + Location: 7 March (Monday), Orchard View Room
Abstract: Inverse problems are ubiquitous. People probe the media with sources and measure the outputs. At the scale of quantum, classical, statistical and fluid, these are inverse Schroedinger, inverse Newton’s second law, inverse Boltzmann problem, and inverse diffusion respectively. The universe, however, should have a universal mathematical description, as Hilbert proposed in 1900. In this talk, we present a line of research results that unify all these inverse problems. Facing the IFDS crowd, I’d like to ask the following question: how to integrate the mathematical equivalence into the optimization formulation for a more efficient algorithmic pipeline than the traditional PDE-constrained optimization?
Bio: Qin Li is an associate professor of mathematics at UW-Madison. Her research lies between scientific computing and PDE-constrained optimization.

IFDS Ideas Forum
2022-03-07 14:00 (America/Chicago)
Multiscale inverse problem, from Schroedinger to Newton to Boltzmann

SILO: Ulugbek Kamilov
2022-03-09 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States

ML-Opt @ UWash: Sean Welleck
2022-03-11 13:30 - 14:30 (America/Los_Angeles)
Speaker: Sean Welleck
Title: Constrained text generation through discrete and continuous inference
Abstract: Neural text generation has shifted to generating text with large-scale, general-purpose models coupled with generic inference algorithms. An important open question is how to efficiently offer control over generated text. We describe two algorithms for enabling control through inference-time constraints: (i) A*-Neurologic, a discrete search algorithm for incorporating logical constraints through estimates of the future, and (ii) COLD decoding, which treats generation as continuous gradient-based sampling from an energy function that captures task-relevant constraints. Our algorithms can be applied directly to off-the-shelf models without the need for task-specific finetuning, and result in strong performance on a variety of generation tasks.

IFDS Ideas Forum: How to Make the Gradients Small in Convex and Min-Max Optimization
2022-03-21 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States
Speaker: Jelena Diakonikolas

IFDS Ideas Forum
2022-03-21 14:00 (America/Chicago)
How to Make the Gradients Small in Convex and Min-Max Optimization

SILO: Gitta Kutyniok
2022-03-23 12:30 - 13:30 (America/Chicago) | Orchard View Room, 330 N. Orchard Street, 3rd Floor NE, Madison, Wisconsin, 53715, United States
