A Pipelined Point Cloud Based Neural Network Processor for 3-D Vision With Large-Scale Max Pooling Layer Prediction
Dongseok Im, Donghyeon Han, Sanghoon Kang, Hoi‐Jun Yoo · IEEE Journal of Solid-State Circuits · 2021
The point cloud data provides useful geometric information to 3-D intelligent systems such as autonomous driving, 3-D reconstruction, and hand pose estimation (HPE). Many mobile devices have implemented the 3-D intelligent system with their limited hardware resources. However, previous processors were not designed for accelerating the point cloud based neural network (PNN) which consists of sampling-grouping layers (SGLs) and convolution layers (CLs). In this article, a pipelined PNN processor is proposed for low latency PNN-based 3-D intelligent systems in mobile devices. The processor adopts the pipelined heterogeneous architecture to accelerate both SGLs and CLs in PNNs. The window-based sampling-grouping (WSG) algorithm boosts up the throughput$\times $2.34 higher in SGLs by directly sampling and grouping the 3-D point cloud data from the depth image. Furthermore, the max pooling (MP) prediction core (MPPC) predicts the large-scale (64- and 128-to-1) MP layers, which increases the throughput by$\times $1.31 higher. Besides, the MP prediction on the tiled data can hide the latency of the MPPC and solve the bank conflict problem on the in-out memories in the convolution core (CC). As a result, the processor successfully demonstrates the PNN-based HPE system resulting in 4.45 ms of the processing time with 8.24 mm of HPE error and 266 mW of power consumption.