Please use this identifier to cite or link to this item:
https://hdl.handle.net/2440/129989
Citations | ||
Scopus | Web of Science® | Altmetric |
---|---|---|
?
|
?
|
Type: | Conference paper |
Title: | Watch, reason and code: Learning to represent videos using program |
Author: | Duan, X. Wu, Q. Gan, C. Zhang, Y. Huang, W. Van Den Hengel, A. Zhu, W. |
Citation: | Proceedings of the 27th ACM International Conference on Multimedia (ACM Multimedia 2019), MM '19, 2019, pp.1543-1551 |
Publisher: | Association for Computing Machinery |
Publisher Place: | online |
Issue Date: | 2019 |
ISBN: | 1450368891 9781450368896 |
Conference Name: | 27th ACM International Conference on Multimedia (ACM Multimedia) (21 Oct 2019 - 25 Oct 2019 : Nice, France) |
Statement of Responsibility: | Xuguang Duan, Qi Wu, Chuang Gan, Yiwei Zhang, Wenbing Huang, Anton Van Den Hengel, Wenwu Zhu |
Abstract: | Humans have a surprising capacity to induce general rules that describe the specific actions portrayed in a video sequence. The rules learned through this kind of process allow us to achieve similar goals to those shown in the video but in more general circumstances. Enabling an agent to achieve the same capacity represents a significant challenge. In this paper, we propose a Watch-Reason-Code (WRC) model to synthesise programs that describe the process carried out in a set of video sequences. The ‘watch’ stage is simply a video encoder that encodes videos to multiple feature vectors. The ‘reason’ stage takes as input the features from multiple diverse videos and generates a compact feature representation via a novel deviation-pooling method. The ‘code’ stage is a multi-round decoder that the first step leverages to generate a draft program layout with possible useful statements and perceptions. Further steps then take these outputs and generate a fully structured, compile-able and executable program. We evaluate the effectiveness of our model in two video-to-program synthesis environments, Karel and ViZdoom, showing that we can achieve the state-of-the-art under a variety of settings. |
Keywords: | video understanding; video embedding; video to program translation |
Rights: | © 2019 Association for Computing Machinery. |
DOI: | 10.1145/3343031.3351094 |
Grant ID: | http://purl.org/au-research/grants/arc/DE190100539 |
Published version: | https://dl.acm.org/doi/proceedings/10.1145/3343031 |
Appears in Collections: | Aurora harvest 4 Computer Science publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
hdl_129989.pdf | Accepted version | 1.71 MB | Adobe PDF | View/Open |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.