Integrated web resource usage monitoring
Alexandros C. Dimopoulos, Martin Reczko · 2018
The Greek ELIXIR node employs two different web analytics services to monitor national web-services: Matomo [1] (formerly Piwik) and Google Analytics (GA). Both services are well known and established, offering the ability to create comprehensive usage reports and are expected to be fully GDPR compliant. Although the produced reports share similar philosophies, they are not identical and hence are not directly comparable. We implemented an automated pipeline in order to seamlessly collect data from both services and present them in a fully comparable way. The final report includes the monitoring data from both GA and Matomo and is produced from template LaTeX code [2], with inline R commands [3]. The raw data are downloaded differently from each web analytics service. For GA, only reports are presented and stored as pdf or image files since direct access to the raw data is not possible; however, Google offers an API [4] for querying data. The Google Analytics [5] add-on for GoogleSheets (based on the particular API) is used, allowing the storage of specific metrics in GoogleSheets spreadsheets. In the case of Matomo, an API is used to store all raw data locally as Excel (xlsx) files. The GoogleSheet and Excel spreadsheets are preprocessed, so that only necessary data are kept, similar metrics are accumulated and the processed data are stored into different tabs of a single Excel spreadsheet. Finally, at LaTeX compile time along with the text, the portions of R code are evaluated and based on the data stored on the single Excel spreadsheet, plots and tables are created directly into a comprehensive report. Combining the major open and closed source monitoring services into a common report offers a flexible resource usage observation method for all ELIXIR nodes. The project is supported by ELIXIR-GR [1] https://matomo.org [2] https://www.latex-project.org [3] https://yihui.name/knitr [4] https://developers.google.com/analytics/devguides/reporting/core/dimsmets [5] https://chrome.google.com/webstore/detail/google-analytics/fefimfimnhjjkomigakinmjileehfopp?utm_source=permalink