Multi-Locality Based Local and Symbiotic Computing for Interactively fast On-Demand Weather Forecasting for Small Regions, Short Durations, and Very High-Resolutions
Bård Fjukstad · NORA - Norwegian Open Research Archives · 2014
Modern weather forecasting is based on forecasts from huge numerical models executed on large supercomputer clusters in large computer centers. Production deadlines for forecast and available computational resources are the main factors for having to limit the forecast region size, spatial resolution, forecast duration and detail level of the weather model used. Modern numerical models currently used by meteorological services exhibit characteristics of weak scaling. The time to produce a forecast of fixed size decreases less and less with increasing number of CPU cores used. Disregarding communication costs, will how fast a computer cluster can compute a forecast be limited by how fast a single core on a node can compute one time-step for a single mesh point in the model. Adding more CPU cores allows a forecast for a larger region to be computed within the same time period. Forecasts for a small region can be computed using few CPU cores as fast or faster than a large region using many cores. The publicly available weather forecast from national weather services as well as from other public services has a set of characteristics. The forecasts cover large regions with a resolution not high enough to include important features and detailed flow patterns in regions with complex terrain. Excepting the National Weather Service in the USA, the services make public only a small subset of available forecast parameters. Forecasts for large areas are computed using batch systems at fixed intervals using whatever observations are available at the fixed starting times. When a user request a forecasts, the latest forecast may typically be up to 6 - 12 hour old. It is not computed on-demand when the user requests the forecast. Visualizations from weather services are often from a static set of types, styles and features and are created for a web browser or apps on commodity devices. Most visualization are not suitable for specialized devices, like a display wall or a smart watch. In this dissertation, tradeoffs are made in model parameters to identify the highest resolution being meteorologically sound and still computable on commodity hardware. The dissertation identifies also the benefits resulting from combining multiple overlapped very high-resolution forecasts. The architecture, design and implementation for commodity platforms for two systems are described. The first system is for computing on-demand very high-resolution interactively fast forecasts. Forecasts can be produced with a resolution better than what is publicly available, for small regions and with short duration, on modern personal desktop computers. The forecasts can be produced for any location on the globe, with a user specified resolution. The second system is for combining local weather forecasts done by multiple users with overlapping regions. The forecasts can be shared between neighbors using a developed collaborative system. Combining shared forecasts enables estimation of forecast uncertainty, thereby increasing the value of the individual forecast. The combined forecasts also allows for computing a measurement of trust between peers. The produced forecast allows for individual visualization, and may contain all possible parameters and vertical levels from the model. A set of experiments was conducted on the prototype systems to document their characteristics. By reducing the problem size, a numerical forecast computation can be moved from a supercomputer to a personal computer. This dissertation documents that the forecasts are produced faster, on-demand, at a higher resolution and are meteorologically sound. The forecasts can realistically be produced on-demand for any location on the globe. This dissertation presents the principle of short traveled data and computations. One realization of this principle is locally produced weather forecasts. The produced forecasts have a local applicability, are computed and stored locally and have only a very limited dependency on third parties and external networks. A set of computational models are described including forecast uncertainty estimation, embarrassingly distributed computations, symbiotic collaborative weather forecasting, forecast visualization using augmented reality and trust establishment using local data. The dissertation has a number of concussions including; A modern numerical model can be executed with at least 1 km resolution for a limited area providing forecasts with details not available in other forecasts today. Combining multiple overlapping very high-resolution forecasts provides information usable for error or uncertainty estimation. The architecture utilizes the locality principle and the short traveled computations and data principle. Experiments show that the typical execution times range from 2 minutes for a 10 km forecast to 15 minutes for a 1 km forecast computed using a quad-core Intel processor with hyper-threading enabled. The architecture for the collaborative symbiotic weather forecast systems is functional without requiring prior-knowledge of peers. All peers are found on-demand, scaling well because of the limited number of peers needed. The simplicity of exchanging forecasts also demonstrates that given these constrains, a fully decentralized exchange of forecast is viable in the sense that it will lead to forecasts that are more useful. While the simple way of finding peers succeeds in localizing all possible peers in seconds in a laboratory setting, no experiments have been conducted outside the laboratory.