RSVQA: Visual Question Answering for Remote Sensing Data

Sylvain Lobry*, Diego Marcos, Jesse Murray, Devis Tuia

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

148 Citations (Scopus)

Abstract

This article introduces the task of visual question answering for remote sensing data (RSVQA). Remote sensing images contain a wealth of information, which can be useful for a wide range of tasks, including land cover classification, object counting, or detection. However, most of the available methodologies are task-specific, thus inhibiting generic and easy access to the information contained in remote sensing data. As a consequence, accurate remote sensing product generation still requires expert knowledge. With RSVQA, we propose a system to extract information from remote sensing data that is accessible to every user: we use questions formulated in natural language and use them to interact with the images. With the system, images can be queried to obtain high-level information specific to the image content or relational dependencies between objects visible in the images. Using an automatic method introduced in this article, we built two data sets (using low- and high-resolution data) of image/question/answer triplets. The information required to build the questions and answers is queried from OpenStreetMap (OSM). The data sets can be used to train (when using supervised methods) and evaluate models to solve the RSVQA task. We report the results obtained by applying a model based on convolutional neural networks (CNNs) for the visual part and a recurrent neural network (RNN) for the natural language part of this task. The model is trained on the two data sets, yielding promising results in both cases.
Original languageEnglish
Pages (from-to)8555-8566
JournalIEEE Transactions on Geoscience and Remote Sensing
Volume58
Issue number12
DOIs
Publication statusPublished - Dec 2020

Keywords

  • Convolution neural networks (CNNs)
  • data set
  • deep learning
  • natural language
  • OpenStreetMap (OSM)
  • recurrent neural networks (RNNs)
  • very high resolution (HR)
  • visual question answering (VQA)

Fingerprint

Dive into the research topics of 'RSVQA: Visual Question Answering for Remote Sensing Data'. Together they form a unique fingerprint.

Cite this