Drift vs Shift: Decoupling Trends and Changepoint Analysis

Drift vs Shift: Decoupling Trends and Changepoint Analysis (via Zoom)

Abstract: Distinguishing between global or macro patterns and local or micro fluctuations helps summarize the evolution of complex non-stationary dynamic systems. Herein, we focus on making distinctions between drift and shifts. Drift describes the micro-level evolution of a process. This may appear as variation about gradual trends. In contrast, shifts refer to discontinuities, rapid changes, or major breaks in trend. These represent macro-level changes in a process. Both trends and shifts might be mechanistically or stochastically generated and/or modeled. However, the underlying causes of shifts are typically different from those of drift. While understanding such differences is a prime objective, this first requires distinguishing shifts from drift. 

We introduce a new approach for decoupling trends (drift) and changepoints (shifts) in time series. Our locally adaptive model-based approach for robustly decoupling combines Bayesian trend filtering and machine learning based regularization. An over-parameterized Bayesian dynamic linear model (DLM) is first applied to characterize drift. Then a weighted penalized likelihood estimator is paired with the estimated DLM posterior distribution to identify shifts. We show how Bayesian DLMs specified with so-called shrinkage priors can provide smooth estimates of underlying trends in the presence of complex noise components. However, their inability to shrink exactly to zero inhibits direct changepoint detection. In contrast, penalized likelihood methods are highly effective in locating changepoints. However, they require data with simple patterns in both signal and noise. The proposed decoupling approach combines the strengths of both, i.e. the flexibility of Bayesian DLMs with the hard thresholding property of penalized likelihood estimators, to provide changepoint analysis in complex, modern settings. The proposed framework is outlier robust and can identify a variety of changes, including in mean and slope. It is also easily extended for analysis of parameter changes in time-varying parameter models like dynamic regressions. We illustrate the flexibility and contrast the performance and robustness of our approach with several alternative methods across a wide range of simulations and application examples.

Bio: David S. Matteson is Associate Professor and Associate Chair of Statistics & Data Science at Cornell University, where he is a member of the ILR School, Cornell Bowers Computing & Information Science, the Center for Applied Mathematics, the Field of Operations Research, and the executive committee for the Center for Data Science for Enterprise & Society. Professor Matteson received his PhD in Statistics at the University of Chicago and his BSB in Finance, Mathematics, and Statistics at the University of Minnesota. He received early Career Award's from the National Science Foundation and the Xerox/PARC Foundation. He is the founding Editor-in-Chief for the new `Data Science in Science' journal. He is an elected officer for the Business & Economic Statistics Section of the American Statistical Association. He is coauthor of `Statistics and Data Analysis for Financial Engineering'. He is director of the NSF funded PRISM Institute for Trans-domain Systemic Risk (sites.google.com/view/prism-prj) and co-director of the NSF funded TRIPODS Greater Data Science Cooperative Institute (gdsc.cornell.edu).