Hosting Data Packages via drat: A Case Study with Hurricane Exposure Data
Brooke Anderson G., Dirk Eddelbuettel · The R Journal · 2017
Data-only packages offer a way to provide extended functionality for other R users.However, such packages can be large enough to exceed the package size limit (5 megabytes) for the Comprehensive R Archive Network (CRAN).As an alternative, large data packages can be posted to additional repostiories beyond CRAN itself in a way that allows smaller code packages on CRAN to access and use the data.The drat package facilitates creation and use of such alternative repositories and makes it particularly simple to host them via GitHub.CRAN packages can draw on packages posted to drat repositories through the use of the 'Additonal_repositories' field in the DESCRIPTION file.This paper describes how R users can create a suite of coordinated packages, in which larger data packages are hosted in an alternative repository created with drat, while a smaller code package that interacts with this data is created that can be submitted to CRAN. Motivation"Big data", apart from being a buzzword, also accurately describes the current scale of many scientific data sets.While the R language and environment (R Core Team, 2017a) enables the creation, use, and sharing of data packages to support methodology or application packages, the size of these data packages can be very large.The Bioconductor project has addressed the potentially large size requirements of data packages through the use of Git Large File Storage, with the package contributor covering costs for extremely large data packages (over 1 gigabyte) (Bioconductor Core Team, 2017).The Bioconductor repository, though, is restricted to topic-specific packages related to bioinformatics.The Comprehensive R Archive Network (CRAN), which archives R packages on any topic, has a recommended size limit (reasonable for a widely-mirrored repository) of 5 megabytes (MB) for package data and documentation (R Core Team, 2017b).A twofold need therefore arises for package maintainers seeking to share large R data packages that are outside the scope of the Bioconductor project.First, there is a need to create and share such a data package for integration into and extensions of a given methodology or application package, and, second, there is a need to integrate use of such a package in a way that makes it seamlessly integrated with smaller CRAN packages that use the data package.Here, we outline one possible approach to satisfy these needs by creating a suite of coordinated packages, in which larger data packages are hosted outside of CRAN but can still be accessed by smaller code packages that are submitted to CRAN.