So lately I have been following Udacity’s wonderful course on robotics and in particular was fascinated by the procedure of Kalman filter. My attempt here will be to expound on certain mathematical subtleties and missing deductive steps in the videos.
The first one has to do with updating prediction by incorporating observation. Somehow in my entire graduate education on probability the notion of product of densities functions is de-emphasized, or maybe I was too prematurely drawn into research to take advantage of the vast number of applied courses that do teach this stuff.
Imagine we have two independent observations of the same 1-dimensional quantity x, represented by and , and suppose somehow we also know the variances and . The question now is what will be the best estimate of what x actually is based on these two observations? Here by estimate we can only mean a single number. As a perhaps stupid remark, for single observations, the observed value is the unbiased minimum variance estimate of the mean itself: so the UMV estimator of given is in fact .
Obviously some parametric assumption is needed. For instance one can assume the estimate takes the form for some , that is, the final estimate is a convex combination of the two individual ones. This assumption is in fact more natural than I first thought, since it guarantees the estimator will be unbiased.
So now we have to find a suitable . What property do we desire in the final estimate , treated as a random variable? Why not stipulate that its variance is minimized in addition to being unbiased (i.e., UMV)? This then becomes an interesting optimization problem:
. One can easily solve this to obtain
Plugging in, we also get the following new estimate of the variance:
So what does this have to do with product of Gaussian pdf’s? If you multiply the two Gaussian densities and renormalize, you get exactly the same result as above! First of all, it’s remarkable that the resulting product is again proportional to a Gaussian. The easiest way to see is by multiplying things out:
constant + the exponent of a Gaussian with the above mean and variance.
but conceptually the Fourier transform of a Gaussian is still Gaussian, and convolution of Gaussians is Gaussian, hence the closure extends to products. Note also convolution will increase the variance, but Fourier transform takes a big variance Gaussian to a small variance one, so the renormalized product will get skinnier.
If one thinks about maximal likelihood estimator, if is the true mean, then given two independent observations, and , their joint density is proportional to . The MLE of is precisely (as obtained by optimizing the above density over ). The variance of similarly inherits from those of and , and should coincide with given above. In fact the distribution of will be Gaussian.
p.s. At first I was trying to derive the MVUE from the point of view of Bayesian update, by letting the distribution of be the prior and that of the conditional likelihood function (in this case conditioning doesn’t do anything). I cannot make sense of it. So if someone is an expert in this, please enlighten me!