11. Text File Processing
Society for Industrial and Applied Mathematics eBooks · 2010
11.1 Latitude and DaylightData Acquisition and Conversion11.2 Nearby MillionsWriting and RepresentationSensor technology and storage technology have advanced to the stage where unimaginably large data sets are automatically collected and archived. Aradio telescope may acquire 1012 bytes of information per week during the course of building a digital star map. Equally large volumes of data may be produced from a flight simulation that is used to design a new airliner. Science and engineering in the 21st century is data-driven science and engineering.Working with large data sets involves interaction with files. A common pattern is to (a) read data from a file, (b) compute with it, and (c) write the results to another file. Familiarity with this process is critical.To illustrate these points we consider a pair of file-processing computations. In the first example we use data from the U.S. Naval Observatory to check the accuracy of a simple model that predicts hours of daylight as a function of date and latitude. The daily sunrise/sunset data for a particular city is encoded in a single formatted file. The programming challenge is to convert the information in each file line into numbers that can be subsequently manipulated. Using data to affirm (or undermine) a theory is perhaps the most central activity in all of science.The second example uses data from the 2000 U.S. census to conveniently explore population density across the country. We build a graphical environment that solicits a U.S. location via the mouse and then displays the number of people who live within a specified distance of that location.A fringe benefit of our file-processing discussion is that it builds a sensitivity for the overhead associated with memory access. In the three-step paradigm mentioned above, the overall efficiency typically depends much more on the data motion steps (a) and (c) than upon the actual computations in (b). This is a general fact of life about high-performance computing, discussed earlier in Chapter 8.