Person search by natural language description in Vietnamese using pre-trained visual-textual attributes alignment model
Thi Thanh Thuy Pham, Van-Thanh Nguyen, Hong-Quan Nguyen, Minh-Quan Le, Hoai Phan, Thi-Ngoc-Diep Do, Thuy-Binh Nguyen, Thanh-Hai Tran, Thi‐Lan Le · 2021
Person search by natural language description is a challenging task as it has to model and learn visual-text semantic embedding. While several works have been dedicated to person search by English descriptions, few attempts have been made for other languages. As a result, it lacks of available resource for person search in these languages. Inspired by transfer learning idea in image classification, in this paper, we propose a method for person search by natural language description in Vietnamese using a model whose weights are trained on a large scale dataset for person search in English. To this end, first, the published network architecture of ViTAA that allows to learn effectively the alignment between visual and textual attributes is employed in this work. Then, we propose to apply different techniques for Vietnamese language processing to analyze and extract relevant elements from descriptions in Vietnamese to feed to the network. Finally, to leverage the information learnt from a large scale dataset, we employ model weights trained from a large scale dataset to a dataset dedicated to person search by natural language description in Vietnamese - VnPersonSearch. The obtained accuracies at Top-1, Top-5 and Top-10 for VnPersonSearch are 61.57%, 83.93% and 90.67% respectively. This means that the proposed method can return relevant persons with very high accuracy in first ten results.