A Geometric Characterization of Fisher Information from Quantized Samples with Applications to Distributed Statistical Estimation
Leighton Pate Barnes, Yanjun Han, Ayfer Özgür · 2018
Consider the Fisher information for estimating a vector θ ∈ ℝdfrom the quantized version of a statistical sample X ~ f(x|θ). Let M be a k-bit quantization of X. We provide a geometric characterization of the trace of the Fisher information matrix IM(θ) in terms of the score function Sθ(X). When k = 1, we exactly solve the extremal problem of maximizing this geometric quantity for the Gaussian location model, which allows us to conclude that in this model, a halfspace quantization is the one-bit quantization that maximizes Tr(IM(θ)). Under assumptions on the tail of the distribution of Sθ(X) projected onto any unit vector in ℝd, we give upper bounds demonstrating how Tr(IM(θ)) can scale with k. We apply these results to find lower bounds on the minimax risk of estimating θ from multiple quantized samples of X, for example in a distributed setting where the samples are distributed across multiple nodes and each node has a total budget of k-bits to communicate its sample to a centralized estimator. Our bounds apply in a unified way to many common statistical models including the Gaussian location model and discrete distribution estimation, and they recover and generalize existing results in the literature with simpler and more transparent proofs.