iWISDM: Assessing instruction following in multimodal models at scale (2024)

1 Introduction

A typical day in most people’s lives involves hundreds or thousands of tasks. Most of which are performed without explicit attention. Just in between getting up and getting to work, one may have already performed 5-15 tasks (taking a shower, shaving, making coffee, getting dressed, etc.). Teaching artificial agents to perform similar seemingly mundane tasks has proven to be an extremely difficult computational problem (Konar, 2018). The challenge becomes more apparent when one realizes that each of these seemingly mundane tasks such as making coffee involves tens of steps/actions (Figure 1).The challenge becomes increasingly more significant once we consider more complex tasks such as operating a device or assembling a piece of furniture from its instruction manual. And yet, these tasks are performed proficiently by most individuals in most situations.

Large Language Models (LLMs) have become increasingly capable of comprehending natural language across wide topics and contexts, allowing them to hold meaningful conversations, give expert advice, and analyze data among other features (Brown etal., 2020; Ouyang etal., 2022; Radford etal., 2019). In the meantime, their multimodal counterparts are starting to emerge, signalling broader application of such models across industries. Large Multimodal Models (LMMs) are generally capable of receiving and responding in a range of possible modalities including visual, text, and audio (Alayrac etal., 2022; Liu etal., 2023b; Achiam etal., 2023). For example, the Gemini-Ultra model, which accepts text, image, audio, and video inputs, and responds with a combination of text and image outputs, recently achieved state-of-the-art on a range of single and multimodal benchmarks (Team etal., 2023).

However, existing benchmarks for assessing such models face several shortcomings:(1) Most multimodal benchmarks like FLEURS (audio-based, (Conneau etal., 2023)) and VATEX (video-based, (Wang etal., 2019)) are still unimodal in their inputs, and do not permit detailed assessment of models’ capacity to integrate information across modalities towards task goals. (2) Visual Question-Answering (VQA) datasets like VQAv2 (Goyal etal., 2017) and CLEVR (Johnson etal., 2017) assess reasoning with visual information in static images without addressing temporal information integration and sequential decision-making. (3) Open-ended learning environments such as XLand (Team etal., 2021), Crafter (Hafner, 2021), and Minecraft (Guss etal., 2019) have been utilized for training reinforcement learning agents. It remains unclear whether and how they can be adapted to benchmark LMMs. (4) To the best of our knowledge, none of the existing benchmarks specifically assess models’ ability to precisely follow instructions in the context of decision making tasks, an important measure of reliability and trustworthiness. Despite its importance, conducting such assessments has been particularly challenging because of the difficulty of collecting samples of multi-step tasks with ground truth information. (5) More recent benchmarks such as MME (Fu etal., 2023), Mmbench (Liu etal., 2023c) and MMvet (Yu etal., 2023) cover a wide range of cognitive tasks that start to adopt manually-generated or GPT-powered responses. However, those benchmarks are difficult to scale, which makes them inconvenient when investigating the scaling properties of LMMs. In addition, benchmarks such as OwlEval (Ye etal., 2023) and LVLMeHub (Xu etal., 2023) rely on subjective human responses that often show high variability across individuals.

To close this gap, we designed instructed-Virtual Visual Decision Making (iWISDM), a virtual environment that enables procedural generation of complex, multi-step decision making tasks that test an agent’s capacity to process visual information guided by natural language instructions. iWISDM builds on the compositional nature of natural behaviour (Barker, 1963) and the fact that complex tasks are often compositionally constructed by combining smaller task units in time. We thus developed a framework which allows instantiating visual decision-making tasks as computational graphs that could be combined logically and temporally to construct a virtually infinite number of tasks with varying complexity. The code of iWISDM is available on GitHub at https://github.com/BashivanLab/iWISDM.

2 Related Works

3 Methodology

We developed iWISDM to facilitate the generation of a diverse range of sequential visual reasoning tasks, which vary in complexity and require minimal user intervention. iWISDM encompasses a broad spectrum of tasks that engage executive functions such as inhibition of action, working memory, attentional set, task switching, and schema generalization. These functions are traditionally associated with the prefrontal cortex, a critical area for advanced cognitive processes in the brain (Fuster, 2015; Sun etal., 2023). Notably, the iWISDM task space is designed to accommodate classic working memory and decision-making tasks commonly employed in neuroscience and cognitive science research (Rigotti etal., 2013; Goldman-Rakic, 1992; Fuster, 2009). In the following sections, we begin by detailing the key functionalities of the iWISDM environment, followed by an in-depth description of its design and the implementation of its constituent components.

3.1 Design

Inspired by prior work (Yang etal., 2018), iWISDM generates tasks through a 3-phase procedure: (1) Task graph construction; (2) Node initialization (3) Trial instantiation. Dividing the generation procedure into distinct phases eliminates the necessity of constructing the task graph anew for each task trial. Once all properties associated with the nodes in a given task graph have been specified (phase 2), the user can generate any number of trials of that particular task, each with potentially different stimuli, ground-truth actions, as well as any other inherent stochastic values such as the number of delay frames. In general, iWISDM creates tasks following:

$$\textbf{f},i,\textbf{r}=\texttt{iWISDM}(G)$$

where, G denotes the task graph, f the sequence of visual frames, i the corresponding language instructions, and r the sequence of ground-truth actions for each visual frame within f according to the instruction i.

The task graph G can either be specified by the user or generated automatically via AutoTask. The major distinction between the two resides in the initial task graph construction phase (phase 1). In contrast to the user-specified mode, where the user needs to manually define the task graph, in AutoTask mode, the user needs to only specify the parameters listed in the following section.

3.1.1 Task graph construction

In iWISDM, nodes and edges create directed, acyclic, connected task graphs. Each node represents a predefined operator that contributes to defining the task (each node in an iWISDM graph represents a task operator, so we use the terms ’node’ and ’operator’ interchangeably). Task operators take downstream stimuli/actions as input and output stimuli/actions based on their definitions. While some operators must have parent/upstream operators, root operators define the actions of a task. They form minimal sub-graphs that define sub-tasks (e.g. Get operators are root operators that define the subtask: What is the attribute of an object?). Under user-defined connectivity rules, sub-graphs can be combined to generate corresponding compositional tasks. In our environment, the depth of the graph is measured by the longest path from the root operator to any other operator within the graph.

Each operator has a customizable set of rules that constrain its connections. The specific operators and their permissible connections are described below: (see Figure 3 for visualization):

• •

  Functional Operators: Select: This operator defines stimuli based on three criteria: when, where, and what. “When” refers to the specific frame from which the stimuli originate. “Where” indicates the location of the stimuli within the frame. “What” depends on the particular dataset from which the stimuli are derived. For example, in our ShapeNet environment, the stimuli have three attributes: category (such as car versus plane), identity (which specific car), and view angle (the angle from which the stimulus is rendered). Select operators that have no downstream connections are the terminal nodes of the graph. Conversely, its potential downstream operators may include any Get* functional operator. Switch: Based on the output action of a boolean task, this operator connects the logic to one of two possible paths. Its compulsory downstream connection must be a boolean operator, while its typical upstream connections are subtasks graphs. Get*: This group of operators is responsible for fetching specific properties of stimuli, such as category, location, or identity, exemplified by operators like GetCategory, GetLocation, and GetIdentity. Its direct downstream connection is always a Select operator and its upstream connections can be any boolean operator. CONST: The simplest operator, CONST represents a fixed value. It is often used as a downstream connection for boolean operators that compare attributes.

• •

  Boolean Operators: Exist: Paired with a specific property value (e.g., ’Desk’), this operator poses the question of whether an object with the property (’Desk’) exists. It generates a boolean output and functions as an action generator or downstream connection for boolean operators. And, IsSame, NotSame, Or: These boolean operators process inputs from two boolean operators to produce a boolean outcome. They are critical in constructing logical conditions within the task graph.

With these operators, iWISDM provides two modes for constructing task graphs: user-specified and automatic.

• •

  User-specified task graph construction. In this mode, users have the freedom to fully specify the task graph by manually creating an instance of NetworkX directed graph. Doing so requires a manual definition of all nodes (operators) and edges (connection between operators) within a task graph, thereby establishing the desired task operation logic. Subsequently, the task graph can be fed to the generator function (write_trial_instance) to yield the corresponding task trials.One potential application of this mode is to replicate trials from a specific task such as the n-back or contextual decision-making tasks. In our core build of iWISDM, we have incorporated a collection of classic tasks from neuroscience and cognitive science literature that users can readily access.

• •

  AutoTask graph construction. In AutoTask mode, users can define a custom task space with a set of hyperparameters to procedurally generate tasks. A task space delineates the complexity and permissible operations pertinent to task construction. The available hyperparameters are: (1) number of compositions, i.e. maximum number of Switch operators to compose subtasks (2) each task graph’s maximum depth and maximum number of operators (3) set of operators to sample from.

In addition to the above hyperparameters, in AutoTask mode, the allowed task structure is further constrained by the permitted connectivity for various operators (e.g. And operators must be followed by other boolean operators such as IsSame, NotSame, And, Or). A default operator connectivity is defined for all existing operators in our core build, but new connectivity rules could easily be added for any new user-defined operators. This is done through a Python dictionary, which details the allowed input and output operators for additional operators. By specifying these hyperparameters, iWISDM autonomously generates random runnable task graphs derived from the predefined task space. Each resultant task graph can then be used to generate specific trials.

To ensure each task graph would comply with the connectivity rules between operators, we follow a backward initialization process during AutoTask (Figure A1). The task generation process starts from the root node and descends recursively. For each current node/operator $n$ in the graph, its downstream operators $C_{n}$ are randomly sampled based on the connectivity rules. As the graph depth approaches the specified maximum depth, the permissible operators with the shortest possible subtask depth are sampled into $C_{n}$. For instance in our core build, if $n$ is the And operator, then only IsSame, NotSame are sampled since they have shorter subgraph depths than And, Or. Through this procedure, iWISDM AutoTask facilitates sampling from diverse task spaces with varying degrees of complexity specified by the user. The utilization of the connectivity rule dictionary and Switch operator guarantees the logical and feasible nature of the generated tasks, providing researchers with an extensive pool of tasks for investigation and exploration.

Together, iWISDM’s two operating modes provide users with the flexibility to use the environment to either train or evaluate models on specific tasks (e.g. classic tasks from literature), as well as a wide variety of tasks adhering to specific guidelines as stipulated by the specified hyperparameters.

3.1.2 Node Initialization

The second step assigns values to each node within the task graph, thereby conferring logically coherent tasks. There are two critical challenges: the initialization of an independent task graph and the integration of multiple graphs in time.

To instantiate a task graph (Figure A1a), a backward recursive approach is used (Yang etal., 2018), similar to that of AutoTask task graph generation. During trial instantiation, given the expected stimuli/action output, an operator propagates stimuli-related properties or actions to its children operators in reverse topological order. The process starts from the root operator of the graph that corresponds to the final task action. To guarantee a balanced action space, we uniformly sample from the pool of possible output (e.g. true/false, location values, and object categories) and assign it to each unassigned operator. We then iteratively go one layer down until reaching the Select leaf nodes, propagating the expected output to all nodes. At this stage, each task only has one output action.

The backward process is confined to the instantiation of individual task graphs and ensures logical consistency within each task. To ensure inter-graph logical consistency, we formulated a forward algorithm, generating complex tasks that require output actions at different frames, which we call temporal composition (see section 3.2. During this process, distinct Select operators might assign conflicting attributes to the same stimulus. The forward process (Figure A1b) therefore also serves to resolve disparities between Select operators in the same frame. It identifies the earliest merging frame, and resolves potential conflicts on a frame-by-frame basis, adjusting the properties of ensuing operators.

In summary, the node initialization process focuses on property assignment, involving the initialization of individual graphs followed by fusing them together. This process aims to yield logically coherent task instances, both within subtasks and in temporally merged tasks.

3.1.3 Task trial instantiation

Regardless of the selected operation mode, for each task trial, iWISDM yields a frame sequence, an accompanying natural language instruction, and an action sequence. As delineated above, tasks are initially formulated with graphs that define the task logic. The task graph then serves as the basis for instantiating task trials. Distractors and fixation cues are also added during this step. Each task trial comprises of the following distinct components:

• •

  Frame sequence. The frame sequence consists of an array of images. They display the visual information at each time step of the trial. Each frame is accompanied by a dictionary that contains the object properties within that frame. Images are stored as PNG files.

• •

  Natural language instructions. Each trial is accompanied by a natural language instruction that explains the task steps and decision criteria. Natural language instructions are generated concurrently during task instantiation. A partial string is assigned to each operator in the task graph that depends on its definitions and initialization (see Figure A1 for an example). This approach allows iWISDM to automatically produce contextually relevant natural language instructions that describe the task in a human-readable format for each task.

• •

  Action sequences. Each trial also consists of an array of ground-truth actions at each frame of the trial. The action sequences could be used for supervised training and validation of the agents on the generated trials.

• •

  Distractors (optional) For users interested in making trials more attention-demanding, we offer a pre-implemented solution that adds distractors post-hoc. By specifying parameters during trial generation, distractors can be added without causing conflicts with existing task rules. The major challenge lies in distinguishing between stimuli and distractors in task instructions. This is achieved by specifying the attributes of the stimulus that are not used during task execution. Additionally, it is important to avoid using distractors that share the same non-used attribute values. Based on the instructions, an ideal agent should be able to identify the stimulus of interest accurately. An example can be seen in Appendix Figure A5.

4 Evaluation

4.3 Results

Figure 4a shows the accuracy of actions taken by the models plotted against the complexity level for each type of prompt. GPT-4V generally achieves the best model scores, with the largest performance gap on the low and high complexity tasks. However, relative to human performance these gaps are marginal at best. Broadly, MMICL was the worst-performing model. The expected inverse correlation between complexity and action accuracy was only captured clearly by the GPT-4V and Gemini-Pro-1.0 results.

In contrast to model performance, human subjects scored much more accurately, with scores ranging from 0.78 to 0.98 across complexities. This model-human gap in performance indicates a significant shortcoming of LMMs on multi-image instruction following tasks. This shortcoming is unlikely due to insufficient feature understanding, as the high single-frame category task performance (Figure 4b) conflicts with the observed weak category task performances across complexities.

We also analyzed how each model performed on subsets of tasks where single or multiple object properties were involved. Figure 4 b-e shows the average accuracy of all models conditioned on task properties and complexity level. We did not include the object-identity subgroup in the high complexity plot of Figure 4e as non-boolean response types for object identity tasks are not feasible. The performances on high complexity tasks show a clear ranking between the abilities of GPT-4V and Gemini-pro models on tasks with diverse response options. For almost all models and complexities, location-only tasks posed the most difficulties. This finding confirms previous analyses of GPT-4V, which found that it often struggles to correctly recognize an object’s position within an image¹¹1https://blog.roboflow.com/gpt-4v-object-detection (Majumdar etal., 2024).

To further investigate the response patterns of the LMMs we performed further analyses on a subset of the models seen in Appendix Figures A7, A8, A9, and A10. To determine the effects of delay frames on model performance a set of simple delayed-match-to-sample tasks were generated which only differed in difficulty by the number of delay frames. Appendix Figure A7 shows that for InternLM-XComposer2, MMICL, and GPT-4v, the addition of single or multiple delay frames in a task has little effect on task performance. Next, we looked into how the number of different boolean operators affected open-source model (InternLM-XComposer2 & MMICL) performance. The Appendix Figure A8 a-d results show that, generally, across all boolean operators an increase in their abundance within a task leads to worse model performance, which is expected. In Appendix Figure A9 we examined how the number of stimuli affected task performance across complexities. We expected to see that as the number of stimuli increases, the task performance would decrease. This was found to be the case for Low complexity tasks, which can be seen in Appendix Figure A9 a. However Medium and High complexity tasks displayed an inverse trend, as seen in Appendix Figure A10 b and c. Finally, we investigated the exact effect that different required response types had on accuracy for the High complexity benchmark. As Appendix Figure A10 shows, when tasks required a response that was a non-boolean type word, the models performed significantly worse.

5 Conclusion and future directions

We introduced iWISDM as a platform for validating multimodal models. As a benchmark, our primary focus is on assessing the ability of LMMs to follow instructions in visual-language decision-making tasks.We developed three benchmarks of incremental complexity and evaluated several LMMs and human performance. The gap between LMMs and humans indicates that LMMs still lack key abilities to solve instruction-following tasks. Through a detailed analysis of LMMs’ behaviour patterns, we identified diminished spatial recognition ability and decreased performance with increased task complexity.

We believe iWISDM would be an important benchmark that complements existing benchmarks which evaluate LMM capabilities in areas such as commonsense reasoning, numerical computation, or relational inferences (Fu etal., 2023). Although the current version of iWISDM lacks the capacity to probe these functions, it could potentially cover some of these additional capabilities by adding new operators like Count (to tally specific objects) or Relative (to identify properties like location in relation to other objects). We also believe iWISDM can serve as an important benchmark for continual learning algorithms evaluation. For detailed discussions, please refer to Appendix A.1.1.

Moreover, the currently used stimulus dataset is derived from publicly accessible sources (Chang etal., 2015), which may pose a risk of data leakage. To address this, we plan to introduce a variety of datasets that users can easily select from. Finally, the compositional nature of iWISDM tasks may also provide an avenue for exploring the failure modes of current LMMs by investigating their specific weaknesses and specifically targeting those during training. We are looking to develop detailed evaluation criteria tailored to iWISDM and to establish an evaluation platform, along with a continuously updated leaderboard, for the purpose of measuring and comparing the performances of models.

References

• Achiam etal. (2023)Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, FlorenciaLeoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, etal.Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023.
• Alayrac etal. (2022)Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, etal.Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
• Aljundi etal. (2019)Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.Task-free continual learning.In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11254–11263, 2019.
• Bai etal. (2023)Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023.
• Barker (1963)RogerG Barker.The stream of behavior: Explorations of its structure & content.1963.
• Brown etal. (2020)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, JaredD Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, etal.Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020.
• Chang etal. (2015)AngelX Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, etal.Shapenet: An information-rich 3d model repository.arXiv preprint arXiv:1512.03012, 2015.
• Chen etal. (2022)Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny.Visualgpt: Data-efficient adaptation of pretrained language models for image captioning.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18030–18040, 2022.
• Chen etal. (2023a)Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, and Mohamed Elhoseiny.Video chatcaptioner: Towards the enriched spatiotemporal descriptions.arXiv preprint arXiv:2304.04227, 2023a.
• Chen etal. (2023b)Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793, 2023b.
• Chen etal. (2020)Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed ElKholy, Faisal Ahmed, Zhe Gan, YuCheng, and Jingjing Liu.Uniter: Universal image-text representation learning.In European conference on computer vision, pp. 104–120. Springer, 2020.
• Chiang etal. (2023)Wei-Lin Chiang, Zhuohan Li, ZiLin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, JosephE Gonzalez, etal.Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
• Chung etal. (2022)HyungWon Chung, LeHou, Shayne Longpre, Barret Zoph, YiTay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, etal.Scaling instruction-finetuned language models.arXiv preprint arXiv:2210.11416, 2022.
• Computer-Vision-in-the-Wild (2024)Computer-Vision-in-the-Wild.Cvinw readings: A collection of papers on the topic of “computer vision in the wild (cvinw)”.https://github.com/Computer-Vision-in-the-Wild/CVinW_Readings, 2024.Accessed: 2024-02-16.
• Conneau etal. (2023)Alexis Conneau, Min Ma, Simran Khanuja, YuZhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna.Fleurs: Few-shot learning evaluation of universal representations of speech.In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 798–805. IEEE, 2023.
• Dai etal. (2023)WDai, JLi, DLi, AMH Tiong, JZhao, WWang, BLi, PFung, and SHoi.Instructblip: towards general-purpose vision-language models with instruction tuning. arxiv.Preprint posted online on June, 15:2023, 2023.
• Dasgupta etal. (2023)Ish*ta Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus.Collaborating with language models for embodied reasoning.arXiv preprint arXiv:2302.00763, 2023.
• Deng (2012)LiDeng.The mnist database of handwritten digit images for machine learning research [best of the web].IEEE signal processing magazine, 29(6):141–142, 2012.
• Driess etal. (2023)Danny Driess, Fei Xia, MehdiSM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, etal.Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023.
• Fu etal. (2023)Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, XuLin, Jinrui Yang, Xiawu Zheng, KeLi, Xing Sun, etal.Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023.
• Fuster (2015)Joaquin Fuster.The prefrontal cortex.Academic Press, 2015.
• Fuster (2009)JoaquínM Fuster.Cortex and memory: emergence of a new paradigm.Journal of cognitive neuroscience, 21(11):2047–2072, 2009.
• Goldman-Rakic (1992)PatriciaS Goldman-Rakic.Working memory and the mind.Scientific American, 267(3):110–117, 1992.
• Goyal etal. (2019)Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra.Scaling and benchmarking self-supervised visual representation learning.In Proceedings of the ieee/cvf International Conference on computer vision, pp. 6391–6400, 2019.
• Goyal etal. (2017)Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.Making the v in vqa matter: Elevating the role of image understanding in visual question answering.In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904–6913, 2017.
• Guss etal. (2019)WilliamH Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov.Minerl: A large-scale dataset of minecraft demonstrations.arXiv preprint arXiv:1907.13440, 2019.
• Ha & Schmidhuber (2018)David Ha and Jürgen Schmidhuber.World models.arXiv preprint arXiv:1803.10122, 2018.
• Hafner (2021)Danijar Hafner.Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780, 2021.
• Johnson etal. (2017)Justin Johnson, Bharath Hariharan, Laurens Van DerMaaten, LiFei-Fei, CLawrenceZitnick, and Ross Girshick.Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901–2910, 2017.
• Konar (2018)Amit Konar.Artificial intelligence and soft computing: behavioral and cognitive modeling of the human brain.CRC press, 2018.
• Lai etal. (2023)Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, etal.From scarcity to efficiency: Improving clip training via visual-enriched captions.arXiv preprint arXiv:2310.07699, 2023.
• Lake & Baroni (2023)BrendenM Lake and Marco Baroni.Human-like systematic generalization through a meta-learning neural network.Nature, 623(7985):115–121, 2023.
• Li etal. (2023a)Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan.Seed-bench-2: Benchmarking multimodal large language models.arXiv preprint arXiv:2311.17092, 2023a.
• Li etal. (2023b)Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023b.
• Li etal. (2020)Xiujun Li, XiYin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, LiDong, Furu Wei, etal.Oscar: Object-semantics aligned pre-training for vision-language tasks.In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pp. 121–137. Springer, 2020.
• Lin etal. (2014)Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and CLawrence Zitnick.Microsoft coco: Common objects in context.In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–755. Springer, 2014.
• Liška etal. (2018)Adam Liška, Germán Kruszewski, and Marco Baroni.Memorize or generalize? searching for a compositional rnn in a haystack.arXiv preprint arXiv:1802.06467, 2018.
• Liu etal. (2023a)Haotian Liu, Chunyuan Li, Yuheng Li, and YongJae Lee.Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023a.
• Liu etal. (2023b)Haotian Liu, Chunyuan Li, Qingyang Wu, and YongJae Lee.Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023b.
• Liu etal. (2023c)Yuan Liu, Haodong Duan, Yuanhan Zhang, BoLi, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, etal.Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023c.
• Lomonaco & Maltoni (2017)Vincenzo Lomonaco and Davide Maltoni.Core50: a new dataset and benchmark for continuous object recognition.In Conference on robot learning, pp. 17–26. PMLR, 2017.
• Loula etal. (2018)Joao Loula, Marco Baroni, and BrendenM Lake.Rearranging the familiar: Testing compositional generalization in recurrent networks.arXiv preprint arXiv:1807.07545, 2018.
• Majumdar etal. (2024)Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, etal.Openeqa: Embodied question answering in the era of foundation models.In 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024, 2024.
• Molano-Mazon etal. (2022)Manuel Molano-Mazon, Joao Barbosa, Jordi Pastor-Ciurana, Marta Fradera, Ru-Yuan Zhang, Jeremy Forest, Jorge del PozoLerida, LiJi-An, ChristopherJ Cueva, Jaime dela Rocha, etal.Neurogym: An open resource for developing and sharing neuroscience tasks.2022.
• Ouyang etal. (2022)Long Ouyang, Jeffrey Wu, XuJiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, etal.Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
• Radford etal. (2019)Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, etal.Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019.
• Radford etal. (2021)Alec Radford, JongWook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, etal.Learning transferable visual models from natural language supervision.In International conference on machine learning, pp. 8748–8763. PMLR, 2021.
• Rigotti etal. (2013)Mattia Rigotti, Omri Barak, MelissaR Warden, Xiao-Jing Wang, NathanielD Daw, EarlK Miller, and Stefano Fusi.The importance of mixed selectivity in complex cognitive tasks.Nature, 497(7451):585–590, 2013.
• Saikh etal. (2022)Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya.Scienceqa: A novel resource for question answering on scholarly articles.International Journal on Digital Libraries, 23(3):289–301, 2022.
• Schuhmann etal. (2022)Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, etal.Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
• Shah etal. (2023)Dhruv Shah, Błażej Osiński, Sergey Levine, etal.Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.In Conference on robot learning, pp. 492–504. PMLR, 2023.
• Sun etal. (2023)Weinan Sun, Madhu Advani, Nelson Spruston, Andrew Saxe, and JamesE Fitzgerald.Organizing memories for generalization in complementary learning systems.Nature neuroscience, 26(8):1438–1448, 2023.
• Team etal. (2023)Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, AndrewM Dai, Anja Hauth, etal.Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023.
• Team etal. (2021)Open EndedLearning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, etal.Open-ended learning leads to generally capable agents.arXiv preprint arXiv:2107.12808, 2021.
• Touvron etal. (2023)Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, etal.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023.
• Tsimpoukelli etal. (2021)Maria Tsimpoukelli, JacobL Menick, Serkan Cabi, SMEslami, Oriol Vinyals, and Felix Hill.Multimodal few-shot learning with frozen language models.Advances in Neural Information Processing Systems, 34:200–212, 2021.
• Wang etal. (2019)Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and WilliamYang Wang.Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4581–4591, 2019.
• Xu etal. (2023)Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, YuQiao, and Ping Luo.Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.arXiv preprint arXiv:2306.09265, 2023.
• Yang etal. (2018)GuangyuRobert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathon Shlens, and David Sussillo.A dataset and architecture for visual reasoning with a working memory.In Proceedings of the European Conference on Computer Vision (ECCV), pp. 714–731, 2018.
• Ye etal. (2023)Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, etal.mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023.
• Yu etal. (2023)Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023.
• Zhai etal. (2022)Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.Lit: Zero-shot transfer with locked-image text tuning.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123–18133, 2022.
• Zhang etal. (2021)Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.Vinvl: Revisiting visual representations in vision-language models.In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5579–5588, 2021.
• Zhang etal. (2022)Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, XiVictoria Lin, etal.Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.
• Zhou etal. (2018)Luowei Zhou, Chenliang Xu, and Jason Corso.Towards automatic learning of procedures from web instructional videos.In Proceedings of the AAAI Conference on Artificial Intelligence, volume32, 2018.
• Zhu etal. (2023)Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023.

Appendix A Appendix

A.2 Additional Figures