Enabling Automatic Cinematography with Reinforcement Learning
Zixiao Yu, Chenyu Yu, Haohong Wang, Jian Ren · 2022
Rule-based automated cinematography has proven to be a very effective means of speeding up the filmmaking process and reducing costs by incorporating the rules of many camera shots into the optimization process. However, it is observed that many well-known directors developed their style of lens language and sometimes even violate the conventional rules. It is a very challenging task to exploit these directors' versatile styles into the automatic cinematography framework due to two main reasons: (i) it is not trivial to translate these styles into a library of rules; (ii) the data can be collected from existing films are very limited and the accuracy is insufficient for learning purposes. In this paper, we propose a novel unified Reinforcement learning-based Text to Animation (RT2A) framework that can apply reinforcement learning to automatic cinematography. In RT2A, the decisions by the director on the camera settings are recorded and can be utilized for the future auto cinematography agent training process. A well-designed reward functionality has been proposed that guides the algorithm to find the best policy and mimic the human director's decision-making process for the camera selection of each scene. The experimental results show that the proposed RT2A can effectively imitate the directors' usage of lens language patterns. Compared to the reference algorithm, RT2A can achieve a gain of up to 50% in camera placement acceptance rate and 80% in imitating the rhythm of camera switching.