maxent_toolbox: Maximum Entropy Toolbox for MATLAB, version 1.0.2
Ori Maoz, Elad Schneidman · Zenodo (CERN European Organization for Nuclear Research) · 2017
Maximum entropy toolbox for MATLAB is a free, open-source toolbox for finding the maximum entropy distribution of training data, based on a set of constraints or observables over the data Maximum entropy models give the mathematically minimal probabilistic models of the states or configurations of a systems, given the mean values of some set of observed functions (Jaynes 1957). Since the entropy of a distribution measures the randomness or lack of interaction among different variables (Shannon 1949), the minimally structured distribution given a set of observables is the distribution with the maximal entropy that is consistent with these observables. Mathematically, in its discrete form, if \(x_i\) are the elements of the system (here variables taking discrete values), then the maximum entropy model for \(p(x_1,x_2 \ldots x_n)\) which is consistent with a set of observables of the form \(\langle f_{i}(x_1,\ldots,x_n)\rangle_{p}\) has a unique solution in the form: \(\hat{p}(x_1 \ldots x_n)=\frac{1}{Z} \exp[\sum_i \lambda_i f_i(x)]\) where \(\lambda_i\) are Lagrange multipliers and \(Z=\sum_{x}\hat{p}(x)\). Since this problem is convex, the maximum entropy solution is unique and can be found numerically. This family of models thus offer the minimal model that is consistent with the constraints. This approach has been used as a way to approximate or explore the nature of correlations in systems of many variables ranging from systems of spins, populations of neurons, genes, pixels in images, words in language etc. The current toolbox allows for learning maximum entropy distributions of binary variables \(x_i\in \{0,1\}\) and distributions of patterns of the form 1000110100. The toolbox takes as an input a set of samples of activity patterns and learns a model of the probability over all states, thus extrapolating to the entire distribution over all possible activity patterns. The user can choose between several variants of maximum entropy models, each relying on a different set of observables or constraints. The maximum entropy models currently supported by this package are: Independent model, which uses \(\langle x_i \rangle_{data}\) as constraints Pairwise maximum entropy model: \(\langle x_i \rangle_{data}\) and \(\langle x_i x_j\rangle_{data}\) K-Synchrony model: \(\langle \sum_{i}x_i \rangle_{data}\) K-Pairwise model Arbitrary set of high-order correlations Any combination of the above models The toolbox automatically switches between exhaustive solutions for small (<30) groups of variables and Markov Chain Monte Carlo (MCMC) methods for larger groups and can be used to learn distributions of up to several hundreds of binary variables. The software is provided as an installable toolbox for MATLAB, and most of the code is written in heavily optimized C++ precompiled for Windows (64 bit), OS X and Linux (CentOS). The project is hosted in GitHub: https://orimaoz.github.io/maxent_toolbox/