Developing a System for Free-Form Visual Question Answering

Abstract

In the past few years, we have witnessed significant advances in the field of visual question answering (VQA). This complex task connects the areas of computer vision and natural language processing research to build a step towards solving artificial intelligence (AI). A crucial feature of any AI-complete problem is the ability to scale for real-world applications. For VQA, in particular, it implies that a model should answer any question about any possible image. However, it seems unlikely that any model would be able to learn all the required knowledge from a single training set, so the use of external knowledge has become a promising direction for VQA research. In this thesis, we investigate techniques to exploit external information to improve the performance of visual question answering methods. First, we explore the benefits of unsupervised image pre-training for VQA. We create a dataset of simple images, where only a small fraction is annotated with VQA questions. We experiment with two selfsupervised approaches and show that they can be used for VQA pre-training and generalise well from little annotated data. Next, we frame VQA as a multi-task problem and complement the traditional classification objective with an additional regression loss that aims to learn vector representations of answers. This novel learning branch allows a model to embed prior knowledge about answer semantics, and we show that the information captured in the relations between answer embeddings is important for VQA. This method not only shows clear improvements in accuracy and consistency over a range of different question types but also unlocks the potential for novel answer prediction. Finally, we implement a method that embeds information from external knowledge bases into vision-and-language transformers. This method proposes to optimise an additional objective that aligns learned word representations with the matching knowledge embeddings. We evaluate the applicability of various knowledge bases to multiple downstream tasks and show that the method brings clear improvement on knowledge-demanding and general visual reasoning datasets.