GreaseVision: Rewriting the Rules of the Interface
Siddhartha Datta, Konrad Kollnig, Nigel Shadbolt · 2022
Digital harms can manifest across any interface.Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems.We put forth GreaseVision, a collaborative human-inthe-loop learning framework that enables endusers to analyze their screenomes to annotate harms as well as render overlay interventions.We evaluate HITL intervention development with a set of completed tasks in a cognitive walkthrough, and test scalability with one-shot element removal and fine-tuning hate speech classification models.The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions.Our contribution also enables researchers to study the distribution of multi-modal harms and interventions at scale.