Generating Captions of Imagined content from Human Brain Activities Applying An Image Captioning Model
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama · 2022 IEEE 4th Global Conference on Life Sciences and Technologies (LifeTech) · 2022
The study of human mental content is a longstanding topic, but its vague nature has made examination difficult. In this paper, we propose a method for cognitive estimation via image captioning models, which estimates conceived content from brain activity. Through the image captioning model, we estimate mental content by examining the relationship between data obtained by functional magnetic resonance imaging (fMRI) and semantic information of conceived content. The decoding model based on the regression scheme learned employing stimulus-evoked brain activity in the visual cortex area is able to estimate the content accurately. Finally, our method enables the generation of accurate captions from fMRI activity obtained while the subject is imagining the image. The results of experiments show that our method can generate accurate captions for the conceived content.