Understanding text with an accompanying diagram
William C. Bulko · 1988
Because English is typically not a good language for expressing spatial relationships, people frequently use diagrams to supplement textual descriptions in engineering and scientific applications.BEATRIX is a computer program which takes as input an English statement of a physics problem and a figure associated with it, understands the physical objects described and the relationships between them, and produces a data structure containing a model of the problem.Using a simple graphic interface provided by BEATRIX, the user sketches a picture using a mouse-based menu system similar to other familiar drawing programs.The user may also enter normal English sentences through the keyboard, using the mouse to help in editing.Meanwhile, BEATRIX transparently creates an internal representation of the picture similar to that which might be produced as the output of a vision system.It then translates the text and the picture representation into a semantic network model, resolves the references between objects common to the two sources of knowledge, and produces a unified model of the problem world.The correctness and completeness of this model can be validated when it is supplied as input to a physics problem-solving program which is currently under development.A major problem in the field of knowledge representation is that of coreference: recognizing when two descriptions contain references to the same object or set of objects so that information is associated with the correct object in the internal model.Flexible, opportunistic control is necessary in order to recognize coreference and to act upon it; accordingly, the understanding module of BEATRIX uses a blackboard control structure.The knowledge sources for the blackboard contain information for identifying and matching common objects in physics problems.The blackboard also handles the semantics of parsing the English text and the identification and association of picture elements.It is natural and often necessary for people to communicate using a combination of text and pictures.There are many applications in science and engineering where the ability to communicate in this way with a computer could be used to a great advantage.We believe that BEAT-PdX illustrates a control structure and collection of knowledge that can *This research was supported by the U.