A log file service exploiting write-once storage
Ross Finlayson · 1990
The changing economics of mass storage prompt a reexamination of the way that the computer systems of the future should use permanent storage. In particular, the use of increasingly large main memory caches suggests that most disk traffic will be writes, primarily of long-term, archival data. In addition, processors and main memory are increasing in performance at a greater rate than mass storage devices, making it especially important that the number of seek operations be kept as low as possible. These trends, along with the increased availability of low cost, high-density removable storage media (such as write-once optical disk), make it attractive to treat permanent storage as write-once, and, in particular, append-only. Most file systems, however, are not suitable for write-once storage media. Furthermore, those file systems that have been designed for such media are inefficient for storing files (in particular, logs) that are updated in small increments. The traditional approach of implementing logs merely as files in a standard file system is, therefore, no longer suitable. Instead, we propose that logs should be the fundamental data abstraction provided by an append-only storage medium, with log files being made a available to applications as an operating system service. We describe the design of the logging service, and show how this design provides efficient access to large numbers of independent log files, with low space overhead, despite the restriction on possible data structures imposed by the use of append-only storage. An implementation of Clio for the V-System, using a write-once optical disk for permanent storage, provides measurements in support of these results. Finally, we show how an application may be built upon a logging service such as Clio, with one or more log files providing the application's only form of persistent storage. We describe both the benefits and the drawbacks of this approach, and, in particular, describe how a general file system can be structured in this manner. A trace-driven simulation of such a file system shows that (with moderately large file caches) the overhead introduced by the underlying logging service is very low.