arxiv:2503.14478

Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM

Published on Mar 18

· Submitted by

nebulae09 on Mar 19

Upvote

Authors:

Xinyu Fang ,

Zhijian Chen ,

Shengyuan Ding ,

Xiangyu Zhao ,

Haodong Duan ,

Dahua Lin

Abstract

Creativity is a fundamental aspect of intelligence, involving the ability to generate novel and appropriate solutions across diverse contexts. While Large Language Models (LLMs) have been extensively evaluated for their creative capabilities, the assessment of Multimodal Large Language Models (MLLMs) in this domain remains largely unexplored. To address this gap, we introduce Creation-MMBench, a multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs in real-world, image-based tasks. The benchmark comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous evaluation, we define instance-specific evaluation criteria for each test case, guiding the assessment of both general response quality and factual consistency with visual inputs. Experimental results reveal that current open-source MLLMs significantly underperform compared to proprietary models in creative tasks. Furthermore, our analysis demonstrates that visual fine-tuning can negatively impact the base LLM's creative abilities. Creation-MMBench provides valuable insights for advancing MLLM creativity and establishes a foundation for future improvements in multimodal generative intelligence. Full data and evaluation code is released on https://github.com/open-compass/Creation-MMBench.

View arXiv page View PDF Project page GitHub repository Add to collection

Community

nebulae09

Paper author Paper submitter 1 day ago

Creation-MMBench is a multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs. It features three main aspects:

Comprehensive Creation Benchmark for MLLM and LLM. The benchmark 765 test cases spanning 51 fine-grained tasks. Each case provides an MLLM with images and context, including role, background information, and instructions. To further explore the impact of visual instruction tuning, we transformed Creation-MMBench into a text-only variant, Creation-MMBench-TO, by replacing image inputs with corresponding textual descriptions.
Robust Evaluation Methodology. Creation-MMBench includes carefully crafted instance-specific criteria for each test case, enabling assessment of both general response quality and visual-factual alignment in model-generated content. The Dual Evaluation, and GPT-4o judge model is the evaluation strategy for Creation-MMBench.
Attractive Experiment Insight. The results highlight the current limitations of MLLMs in context-aware creativity and vision-based language generation, offering valuable guidance for future research and development. The comparison results of MLLM and its based LLM indicate that the open-source MLLM, after visual instruction tuning, consistently performs worse compared to the corresponding LLM on Creation-MMBench-TO. More detailed results can be observed in our paper.