In recent years, we have been exploring denoising problems, where a signal x is measured with additive noise, y=x+z. The goal is to estimate x from y. This framework is directly applicable to many areas where signals are measured in a noisy way, including in imaging systems, audio, and denoising medical data. When a statistical characterization of the input x and noise z is available, a Bayesian approach can be used. However, in many problems these statistics are unavailable, and we must use a universal approach that adapts to the data at hand. Below we describe two denoising approaches that can adapt to unknown input statistics.

**Universal denoising based on context quantization and Gaussian mixture learning**:
Our approach is based on a universal denoiser for stationary ergodic inputs that
performs context quantization; this denoiser was proposed by Sivaramakrishnan and
Weissman.
The key idea is to partition the stationary ergodic signal denoising problem
into multiple denoising problems involving subsequences that are conditionally independent
and identically distributed (i.i.d.). This denoiser has been proved to asymptotically achieve the
minimum mean square error (MMSE)
for signals with bounded components. We overcome the limitation of boundedness by replacing the density
estimation approach of Sivaramakrishnan and Weissman with a Gaussian mixture (GM) learning algorithm.
Specifically, a GM model is learned for each
noisy subsequence, and we obtain an estimate of the distribution of the corresponding clean subsequence by
subtracting the noise variance from each Gaussian component of the learned GM model.
This density estimate is used to denoise the subequence with a standard Bayesian technique.

We have applied our universal denoiser within the approximate message passing (AMP) recovery framework for linear inverse problems, which leads to a promising universal recovery algorithm AMP-UD (AMP with a universal denoiser). A block diagram of our approach appears in the following figure.

- Y. Ma, J. Zhu, and D. Baron, "Approximate Message Passing Algorithm with Universal Denoising and Gaussian Mixture Learning," to appear in IEEE Trans. Signal Proc. (pdf, arxiv, tutorial video).
- Y. Ma, J. Zhu, and D. Baron, "Universal Denoising in Approximate Message Passing," Duke Workshop on Sensing and Analysis of High-Dimensional Data, Durham, NC, July 2015 (poster; no paper).
- Y. Ma, J. Zhu, and D. Baron, "Universal Denoising and Approximate Message Passing," presented at Inf. Theory Applications Workshop, San Diego, CA, February 2015 (talk).
- Y. Ma, J. Zhu, and D. Baron, "Compressed Sensing via Universal Denoising and Approximate Message Passing," Proc. 52d Allerton Conf. Communication, Control, and Computing, Monticello, IL, October 2014 (pdf, talk, arxiv).

**Universality within a model class (a.k.a. the power of mixing)**:
To estimate a parametric signal x with unknown parameters from the noisy observations y,
one may first find the best parameters (i.e., via
maximum likelihood (ML)) θ*, and then plug the parameters θ*
into the MMSE estimator, E[x|y,θ*].
This approach is known as a plug-in denoiser.
The plug-in is often useful, especially when the signal dimension is large,
so that θ* is likely to match the signal well.
For low dimensional problems, however, one may not have sufficient data to obtain an accurate
θ*. The problem in the low dimensional setting is that we over-commit
to one parameter.

To address this challenge, we propose a mixture denoiser (MixD), x̂ = ∫ E[x|y,θ] p(θ|y) dθ, which mixes over the MMSE estimators w.r.t. each possible θ, where the mixing probability is the posterior of θ. The following figure compares the excess mean square error (MSE) of MixD beyond that of the plug-in as a function of the signal dimension N, where the input signal is Bernoulli distributed. It can be seen that MixD has lower excess MSE than the plug-in for small N, and the advantage vanishes as N grows. Both MixD and the plug-in approach the MMSE for large N.

- Y. Ma, J. Tan, N. Krishnan, and D. Baron, "Signal Estimation with Mixtures," Inf. Theory Applications Workshop, San Diego, CA, February 2014 (talk, related arxiv technical report).