Integrated Graphical Representations for Development of Programs with Real-world Input and Output
淳 加藤 · Institutional Repositories DataBase (IRDB) · 2014
Programs that use real-world input and output (real-world I/O), including interactive camera-based programs and robot applications, have different development and runtime environments.While integrated development environments (IDEs) reside in the traditional desktop environment, the runtime environment is in the real world.Real-world I/O may apply to specific situations and/or respond to dynamic changes that cannot be represented intuitively by the existing text-based user interfaces of IDEs.Previous efforts to eliminate the gap between the development and runtime environments include Programming by Example (PbE), in which the user demonstrates operations to the system and the system infers and executes a corresponding program.In PbE, the program is specified using the runtime environment.The drawback is that the PbE system does not allow the user to precisely describe the logic of the program.In this dissertation, we coin the term "Programming with Example (PwE)," which describes a hybrid approach combining PbE and text-based programming.It makes use of example data retrieved from the real world during text-based programming.We propose to integrate graphical representations, such as photos and videos, which represent realworld I/O data, into text-based IDEs.In particular, we provide a model of the program that deals with real-world I/O, that is, out = f (in, c), where c describes static parameters provided prior to the execution of the program (i.e., constants), in and out are dynamic input and output provided during the execution of the program (i.e., are variables), and f is the specification of the program (i.e., the functionality).Accordingly, we discuss three types of graphical representation.First, we discuss the use of photos as graphical representations of situations in the real world.As an experimental implementation, we present Picode IDE, which supports the development of posture data processing applications that handle posture information of humans and/or robots.It uses photos to represent the posture, where textual references may not be very intuitive.It allows the programmer to take a photo of the subject to automatically capture the posture information, supporting PwE by retrieving static data from the real world.Photos are shown inline in the source code editor and provide contextual information that facilitates visualization of the posture, providing an intuitive coding experience.Second, we discuss use of videos as a graphical representation of variables where the contents are dynamically updated during the execution of the program.As an experimental implementation, we present DejaVu IDE, which supports the development of interactive camera-based applications.It provides visualization by automatically recording the camera input, intermediate processing results, and window output.It supports PwE to record and utilize the dynamic behavior of the program.The programmer can replay the recorded data using an interface that resembles a video player to visualize the behavior.It also allows the programmer to update all of the output from the program by re-executing it using the recorded input data.Third, we discuss a method to specify the behavior of the program with the help of graphical editing.As an experimental implementation, we present VisionSketch IDE, which supports development of image processing applications, detecting interesting events from videos recorded with a fixed viewpoint.It requires the programmer to specify the video source at the beginning of the development for PwE support.The programmer can specify regions of interest in the example video to narrow down the list of applicable image processing algorithms or to setup parameters of a selected image processing algorithm.These IDE implementations are designed for different target applications.However, each of them supports the entire workflow of the programmer by integrating a graphical representation into a text-based programming environment.Each graphical representation corresponds to I/O data sampled from the real world.While we focus on visual information in this dissertation, there are other types of real-world I/O data, including sound, haptic technology, smell and taste.We foresee that applications that make use of such multi-sensory data will become increasingly important in the future.The development of programs that deal with real-world I/O inherently benefits from PwE.Therefore, we believe that the findings described in this dissertation will serve as an important foundation to support the development of such applications.論文要旨