TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools
Ashwin Prasad Shivarpatna Venkatesh, Samkutty Sabu, Jiawei Wang, Amir M. Mir, Li Li, Eric Bodden · arXiv (Cornell University) · 2023
GitHub link: https://github.com/secure-software-engineering/TypeEvalPy A Micro-benchmarking Framework for Python Type Inference Tools 📌 Features: 📜 Contains 154 code snippets to test and benchmark. 🏷 Offers 845 type annotations across a diverse set of Python functionalities. 📂 Organized into 18 distinct categories targeting various Python features. 🚢 Seamlessly manages the execution of containerized tools. 🔄 Efficiently transforms inferred types into a standardized format. 📊 Automatically produces meaningful metrics for in-depth assessment and comparison. 🛠️ Supported Tools Supported ✅ In-progress 🔧 Planned 💡 HeaderGen Intellij PSI Llama 2 Jedi Pyre ChatGPT Pyright PySonar2 HiTyper Pytype Scalpel TypeT5 Type4Py 🏆 TypeEvalPy Leaderboard Below is a comparison showcasing exact matches across different tools, coupled with top_n predictions for ML-based tools. Rank 🛠️ Tool Top-n Function Return Type Function Parameter Type Local Variable Type Total 1 HeaderGen 1 186 56 322 564 2 Jedi 1 122 0 293 415 3 Pyright 1 100 8 297 405 4 HiTyper 135 163173175 273737 179225229 369435441 5 HiTyper (static) 1 141 7 102 250 6 Scalpel 1 155 32 6 193 7 Type4Py 135 39103109 193131 99167174 157301314 (Auto-generated based on the the analysis run on 20-10-23 14:51) 🐳 Running with Docker 1️⃣ Clone the repo git clone https://github.com/ashwinprasadme/TypeEvalPy.git 2️⃣ Build Docker image docker build -t typeevalpy . 3️⃣ Run TypeEvalPy 🕒 Takes about 30mins on first run to build Docker containers. 📂 Results will be generated in the results folder within the root directory of the repository. Each results folder will have a timestamp, allowing you to easily track and compare different runs. docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v ./results:/app/results \ typeevalpy 🔧 Optionally, run analysis on specific tools: docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v ./results:/app/results \ typeevalpy --runners headergen scalpel 🛠️ Available options: headergen, pyright, scalpel, jedi, hityper, type4py, hityperdl