Exploring and Exploiting High-dimensional Phenomena in Statistical Learning and Inference

Yue Lu, Harvard John A. Paulson School Of Engineering And Applied Sciences
Pragya Sur, Harvard Faculty of Arts and Sciences
Subhabrata Sen, Harvard Faculty of Arts and Sciences

The massive datasets being compiled by our society present both great challenges and opportunities for the fields of statistical learning and inference. On the one hand, we are challenged by the "data deluge"—the explosive growth of dimensions in modern datasets often surpasses our ability to extract meaningful insights into the problems under study. Many classical learning and inference methods completely fall apart, as the assumptions they are built on do not take into account the very high-dimensional nature of modern data. On the other hand, unique geometric and probabilistic phenomena, including scaling limits, phase transitions, and universality, emerge in high-dimensional settings. A deeper understanding and clever exploitation of such fascinating (and sometimes counter-intuitive) high-dimensional phenomena can translate to both theoretical breakthroughs and novel algorithms that surpass the current state-of-the-art. This research topic lies squarely at the intersection of multiple disciplines—statistics, mathematics, computer science, statistical physics, and signal processing. Although this area has progressed rapidly over the past few years, unfortunately, the tools introduced within the different research communities have not yet fully percolated beyond their disciplinary boundaries. This workshop seeks to initiate a cross-disciplinary conversation by bringing together a diverse group of researchers from different communities. Our goals are to develop a deep understanding of the underlying mechanisms behind the emergence of high-dimensional phenomena, exploit their ramifications for statistical learning and inference, identify the limits of the current collective knowledge, and help shape future research in this area.