Differentiating Functions of the Jacobian with Respect to the Weights
Gary William Flake, Barak A. Pearlmutter · Maynooth University ePrints and eTheses Archive (Maynooth University) · 1999
For many problems, the correct behavior of a model depends not only on its input-output mapping but also on properties of its Jacobian matrix, the matrix of partial derivatives of the model's outputs with respect to its inputs. We introduce the J-prop algorithm, an efficient general method for computing the exact partial derivatives of a variety of simple functions of the Jacobian of a model with respect to its free parameters. The algorithm applies to any parametrized feedforward model, including nonlinear regression, multilayer perceptrons, and radial basis function networks. 1 Introduction Let f(x; w) be an n input, m output, twice differentiable feedforward model parameterized by an input vector, x, and a weight vector w. Its Jacobian matrix is defined as J = 2 6 4 @f1 @x1 @f1 @xn . . . . . . . . . @fm @x1 @fm @xn 3 7 5 = df(x; w) dx : The algorithm we introduce can be used to optimize functions of the form E u (w) = 1 2 J T...