VideoTime3: A 40-uJ/frame 38 FPS Video Understanding Accelerator With Real-Time DiffFrame Temporal Redundancy Reduction and Temporal Modeling
Miaorong Wang, Yujun Lin, Zhekai Zhang, Ji Lin, Song Han, Anantha P. Chandrakasan · IEEE Solid-State Circuits Letters · 2023
VideoTime3 is an accelerator for state-of-the-art video understanding with deep learning on the edge. Different from prior work, it highlights a real-time DiffFrame convolution achieving 2.2x DRAM access reduction compared to conventional convolution, a sorter-free architecture for efficient sparse output stationary dataflow, temporal modeling capability achieving high accuracy on video understanding applications, and optimized data buffering to remove DRAM traffic overhead for temporal modeling and reduce 55%–79% input activation DRAM traffic in depth-wise layers. The chip consumes 40uJ/frame with 38 frames/second at 0.6V in 28nm CMOS.