An Idiosyncratic Journey Beyond Mean Field Theory
Jonathan S. Yedidia · The MIT Press eBooks · 2001
I try to clarify the relationships between different ways of deriving or correcting mean field theory, and present ”translations” between the language of physicists and that of computer scientists. The connecting thread between the different methods described here is the Gibbs free energy. After introducing the inference problem we are interested in analyzing, I will define the Gibbs free energy, and describe how to derive a mean field approximation to it using a variational approach. I will then explain how one might re-derive and correct the mean field and TAP free energies using high temperature expansions with constrained one-node beliefs. I will explore the relationships between the high-temperature expansion approach, the Bethe approximation, and the belief propagation algorithm, and point out in particular the equivalence of the Bethe approximation and belief propagation. Finally, I will describe Kikuchi approximations to the Gibbs Free energy and advertise new belief propagation algorithms that efficiently compute beliefs equivalent to those obtained from the Kikuchi free energy. To appear as a chapter in “Advanced Mean Field Methods Theory and Practice”, eds. D. Saad and M. Opper, MIT Press, 2000. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Information Technology Center America; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Information Technology Center America. All rights reserved. Copyright c Mitsubishi Electric Information Technology Center America, 2000 201 Broadway, Cambridge, Massachusetts 02139 Publication History:– 1. First printing, TR-2000-27, June 2000 ! # %$& ' () * ,+ .-0/ 1 2 314#57698#5 4;:= A@CB#DEB#DE5 F < F G 4,69H 3IB J#KL6MDE314 NPORQTS UWV)XYS'ZL[7Q]\_^`NAabUdcec QT^]fgQ]hiXjcWZL^]UlkmfnQTS \o^T\9cdZLQTUdhpO VTS Ud[ Vrq1\jQsat\_\9Ovu7Ulw,\_^]\_O Q atZxf7V)hLkyu7\9^TUdzCUeO {nhp^|Xjh ^T^]\9XjQTUdO {i}|\9ZpO~ \9cduvQTS \9hp^]fp S \y\MXj\_})q'\9^0Mp Ns nathp^]7VS h [ih O!]yu7z ZpO Xj\Mun\MZLO#Ue\9cduR\_QTS h7u V]VT X_X_\_\9u \9u O UWXj\_cdf UeO q ^]UeO { UeO {Q]hp{p\_QTS \9^A[ SCfCVTUWXjUWVsQYVZpO u Xjhp}|[ QT\_^rV]XjUd\_O QTUWVQ]V9 abS h0O h atZ u Zxf7V hLkQT\9O0athp^]|hpO0[ ^T\MXjUWV\9cefQ]S \rVTZp}\[ ^]hpq cd\_}V9 q 7QXjhp}|\QTh)QTS \MV\[ ^]hpq cd\_}V abUlQ]S%u7Uew1\9^T\9O Qn['\9^]VT['\MXQ]Uez \9V9y}|\jQ]S h7u V_O Zp}\MViZpO u O hLQYZ Q]Ueh O V_Chp}|\hpk QTS UWVXYS Zp[7QT\9^bUdVtQ]S \_^]\jkmh ^T\ru7\9zphLQ]\9uQTh|[ ^]\9VT\_O QTUdO {QT^YZLO'VcWZ Q]Ueh O Vtq'\_Qsa \9\_O0QTS cdZpO {p Zp{p\0hLkyQ]S \[ SCf7VUWXjUWVsQZLO'uvQ]S \cdZpO {p Zp{p\0hLkQ]S \Xjhp}|[ QT\_^oV]XjUd\_O Q]UdVQ9 ZLceQTS h {pSvN`Zp}V ^]\oQ]S Z Q}fRhp^]Ud{pUdO ZLc QT^YZLUdO UeO {nZpV)Zg[ SCf7VTUdX_UdVQ)abUdcec VTS h a QTS ^]hp { SI N abUecdc hpO cefX_h zp\9^#}|\_QTS h7u V?Q]S Z Q N#S Zxzp\t[1\_^YVh O ZLcdcefr'V\Mu pVThQ]S UWV XYS ZL[7Q]\_^ u7hC\9VO hpQZ QTQT\_}|[7QQThq1\Z!QTS h ^Th {pSVT ^Tz \_f.hpk UeQ]VV q is\9XQM)¢b\MZpu7\9^]V UeO QT\9^T\MVsQ]\9uRUdOv}|hp^]\oq Z XY { ^Th O u hpORQ]S \0VQ]ZLQTUWVsQ]UdX9ZLc [ SCfCVTUWX_V`hpkyu7UWVh ^]u7\9^T\Mu Vf7VQT\_}V £m[ Zp^Q]UdX_ cWZL^]cef¤abUeQTS^]\_{ ZL^Yu¥QThQTS QT\MXYS O UW¦ hpk)Zxzp\9^]Zp{pUdO {Rh zp\_^ u7UdVThp^Yu7\9^ VUdO {rQ]S \^]\_[ cdUWX_Z}|\jQTS hCu'§ }|Ue{ S Q ZLcWVhra ZLO Q QTh`X_hpO VT ceQ ^]\jkm\_^]\_O'Xj\9V £s9 §)£ L© §ZpO u£Ea M§abS Udce\RQ]S h VT\vUdO Q]\_^]\9VQT\Mu;UdO QTS \Xjh }[ 7QT\9^iVTX_Ue\9O Xj ceUeQT\9^]ZLQT ^]\hpO{ ^]Zp[ S UWX_ZLc }h7u7\9cdV}|Ue{ S QX_hpO VT ceQ^]\jkm\_^]\_O'Xj\9V£* La §t£spM§`ZLO'u £ «L§j S \X_hpO O \9XQ]UeO {Q]S ^]\9Zpuq1\jQsat\_\9O)Q]S \u7Uew1\9^T\9O Q=}\_QTS h7u V u7\9V]Xj^]Ueq1\9uS \9^T UdV QTS \r¬Udq q V km^T\9\y\_O \_^]{pfpCbkQ]\_^tUeO Q]^Th7u7 X_UeO {`QTS \UdO7km\_^]\_O'Xj\y[ ^Th q ce\9}a \yZp^T UeO QT\9^T\MVsQ]\9uUeO ZLO ZpcefC_UdO {' N=abUdcdcIu7\j~ O \yQ]S \`¬Ueq q'V km^]\_\\_O \_^]{pfpCZLO'uu7\MVTX_^TUdq' S h aQThu7\9^TUdzp\Z}|\MZLOn~ \_cWunZp[ [ ^]hx®7Ue}Z Q]Ueh OQThUeQA'VUdO {Zz ZL^]UWZ QTUdhpO'ZLc?ZL[7 [ ^Th ZpXYSIMNIabUdcecCQ]S \_O\j®7[ cWZLUdOS h avhpO \t}|Ue{ S Q#^]\j Pu7\_^]Udzp ZpO uX_hp^]^T\MXQ QTS }|\9ZpO ~ \_cWu)ZpO u gkm^]\_ \_O \_^]{pUd\9V VUdO {AS Ud{pSQT\_}|[1\_^YZ Q] ^T \_®C['ZLO VTUeh O V abUlQ]SX_hpO7 VsQ]^]ZpUeO \MuhpO \j °O h7u7\q1\_cdUe\_kEV_ NabUecdc \j®7[ cdhp^]\QTS \^]\_cWZ Q]Ueh O VS Ue[ Vq'\_Qsa \9\_OQTS S Ue{ S7 QT\_}|[1\_^YZ Q] ^T\i\j®7[ ZLO'VUdhpO±Zp[ [ ^]h ZpXYS QTS \R2t\jQ]S Zp[ [ ^]hx®CUd}Z Q]Ueh OI ZLO'u QTS \Aq1\_cdUe\_k#[ ^]hp['ZL{ ZLQTUdhpOZpce{ hp^]UlQ]S }7ZLO u['h UeO Qbhp7QbUdO0['ZL^TQTUWXj cWZL^tQTS \\M¦ Uez ZLcd\_O X_\bhLk1QTS \y2t\_QTS \yZp[ [ ^]hx®7Ue}Z Q]Ueh O)ZpO u|q'\9ceUd\jk [ ^]hp['ZL{ ZLQTUdhpOI #UdO ZLcdcef pN#abUecdc u7\9V]Xj^]Ueq1\o3AUd 'XYS U ZL[ [ ^Thx®7Ud}Z QTUdhpO'VQThQTS \0¬Ueq q V ^T\9\\_O \9^T{ fnZLO'u Z u7zp\9^ QTUWV\0O \_a q1\_cdUe\_k[ ^]hp[ Zp{ Z Q]Ueh O Zpce{ hp^]UlQ]S }VrQ]S Z Q\jμoXjUd\_O QTcdfRXjhp}|[ QT\q1\_cdUd\jkEV \9¦ UdzxZpce\9O QtQ]h|QTS h V\Ah q7Q]ZpUeO \Mu0km^Th } QTS \`3AUdC XYS U,km^]\_\A\_O \_^]{pfp ¶A· 11o »s1⁄4*1⁄4 ¶r1⁄2]3⁄4*? AYA#ÂjAYAAAAjAYAAjA9AEsCxEEAEEME EII1'1⁄2YI» AE