Measuring the amount of information in textual messages
Jezekiel Ben-Arie, Qingquan Wu · 2005
Information resides not in words or symbols but in their meaning. When someone says that a document has a lot of information it does not mean that the document has a lot of words. It means that the document provides a lot of meaningful information. But how much information? So far, classical information theory was designed to treat messages as sets of symbols such as alphanumeric symbols. Therefore, it can provide only quantitative measures of information that are related to the probability distributions of such symbols in messages. But not related to their actual amounts of new information. In this paper, we develop a novel information theoretic approach that will enable to measure the quantities of meaningful information in messages base upon the distributions of their real world states, where each message describes a specific set of such states. If one could measure the Amount of Information (henceforth denoted by AoI) it will create a myriad of opportunities to develop new AoI based techniques in information integration that will enable to generate highly concentrated summaries from disparate sources. Our survey of the scientific literature reveals that such an AoI measure for messages was not developed yet. In this paper, we establish a novel information theoretic approach, which represents each textual message by a set of informative terms that could have different states in each message. For example, an informative term such as game score could have many states such as 0:0, 0:1, 2:0 etc. The message is then converted automatically to a Bayesian Tree graph wherein each node represents an informative term or its state or both. The arcs represent the conditional probabilities for a transition of each informative term to the state represented by the next node. We also develop a theoretical foundation that enables to compute the total AoI of each message from these probabilities even if some informative terms are not statistically independent. Keywords - Amount of Information (AoI), Information theoretic approach, Bayesian trees, textual messages, informative terms.