Evaluating FaaS platforms : comparing open-source Function-AS-A-Service platforms in terms of usability, features, documentation, and performance§
Alexander Glück · FH JOANNEUM ePUB · 2024
This thesis focuses on evaluating open-source FaaS systems. There are many open-source FaaS systems available, and testing and evaluating them is time-intensive. This thesis aims to help by looking at popular systems and comparing them based on quantitative and qualitative factors. This is done to help companies looking for a FaaS system find a product that fits their use case quicker without having to evaluate and compare the systems themselves. The thesis also features a brief look at two commercial alternatives in the commercial alternatives section. To evaluate the systems, they were set up and tested in a Kubernetes cluster. The test criteria range from performance testing (i.e., function invocations per second) to comparing qualitative features. The evaluated systems are OpenFaaS, OpenWhisk, Fission, Knative, and Nuclio. These systems were chosen based on their popularity, which was measured in GitHub stars and recent development activity. The performance testing was done by repeatedly invoking a function, measuring the time it took for each function invocation to complete, and comparing the amount of time it took to hit a predefined request limit. Each system's qualitative features were taken from its official documentation and used to compare each system under evaluation. The results indicate no clear winner because each system has certain advantages and disadvantages. By outlining the strengths and weaknesses of each platform, developers can use this data to quickly get an overview of the systems and choose the system that best fits their specific use cases. The results for each system can be found in the evaluation chapter. It can be said that even though commercial offerings advance faster, open-source alternatives deliver the core features required for a FaaS system.