Knowledge Transfer Between Artificial Intelligence Systems
Ivan Tyukin, Alexander N. Gorban, Konstantin Sofeikov, Ilya Romanenko · Frontiers in Neurorobotics · 2018
We consider the fundamental question: how a legacy ``student'' Artificial Intelligent (AI) system could learn from a legacy ``teacher'' AI system or a human expert without re-training and, most importantly, without requiring significant computational resources. Here ``learning'' is broadly understood as an ability of one system to mimic responses of the other to an incoming stimulation and vice-versa. We call such learning an Artificial Intelligence knowledge transfer. We show that if internal variables of the ``student'' Artificial Intelligent system have the structure of an $n$-dimensional topological vector space and $n$ is sufficiently high then, with probability close to one, the required knowledge transfer can be implemented by simple cascades of linear functionals. In particular, for $n$ sufficiently large, with probability close to one, the ``student'' system can successfully and non-iteratively learn $k\ll n$ new examples from the ``teacher'' (or correct the same number of mistakes) at the cost of two additional inner products. The concept is illustrated with an example of knowledge transfer from one pre-trained convolutional neural network to another.