Abstract
Frontier language models are increasingly marketed as "omni" systems capable of perceiving and responding across multiple modalities. However, existing evaluation frameworks predominantly focus on bimodal understanding—typically text paired with one other modality. To address this gap, we introduce the Modality Maturity Index (MMI), a benchmark designed to systematically evaluate the multimodal capabilities of large language models across five modalities—text, image, audio, video, and document—and their combinations, involving up to three modalities in both inputs and outputs.
MMI comprises 893 carefully crafted, self-contained questions, each requiring models to integrate information from multiple input modalities and generate responses that incorporate diverse output formats. Every prompt includes human-authored rubric criteria for each expected output modality. A model's MMI Value represents the average of per-modality scores across all prompts. Since low scores may stem from either failing to generate a modality (lack of presence) or generating incorrect content, we also introduce a supplementary Modality Presence Score (MPS)—a per-prompt F1 score over expected output modalities.
Applying MMI to five frontier multimodal models, we observe MPS values ranging from 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the limited availability of returned modalities for grading, we report MPS as our primary result pending model improvements. To assess the viability of using LLM judges with rubrics for output correctness, we conducted a separate experiment using custom generation tools. On the generated assets, an LLM judge applying the rubrics achieved 70.8% agreement with rubric-blind human annotators, who scored outputs directly without seeing the criteria.
1. Introduction
The rapid evolution of large language models (LLMs) has led to the emergence of "omni" models that promise seamless interaction across text, images, audio, video, and documents. Despite this ambition, current evaluation benchmarks remain narrow, primarily assessing bimodal capabilities—for example, image captioning or audio transcription paired with text. This limitation obscures the true performance of omni models in real-world scenarios where inputs and outputs often span multiple modalities simultaneously.
To bridge this gap, we propose the Modality Maturity Index (MMI), a comprehensive benchmark that evaluates models across five modalities and their combinations. By focusing on multi-modality in both inputs and outputs, MMI provides a more realistic assessment of a model's all-around capability. Our benchmark's design includes:
- 893 self-contained questions covering single and combined modalities (up to three per input/output).
- Human-authored rubrics that define explicit expectations for each output modality.
- Dual scoring metrics: MMI Value (average per-modality scores) and Modality Presence Score (F1 over expected output modalities).
Our experiments with five frontier models reveal significant room for improvement in generating expected modalities, with MPS values below 35% for all tested models. This underscores the urgency of enhancing multimodal generation fidelity. Additionally, our rubric-based LLM judge shows promising alignment with human evaluation, suggesting a scalable path for future benchmarking.
2. Related Work
Existing multimodal benchmarks (e.g., VQA, AudioSet, VideoQA) typically evaluate bimodal understanding, mapping one input modality to a text output. While these provide valuable insights, they fail to capture the complexity of omni models that must handle arbitrary input-output modality combinations. Recent efforts, such as cross-modal transfer tasks, remain limited in scope. MMI differentiates itself by:
- Covering five modalities and up to three-way combinations in both inputs and outputs.
- Providing rubric-based scoring that enables fine-grained assessment.
- Introducing a presence score (MPS) to isolate modality generation failures from content errors.
3. Benchmark Design
3.1 Question Construction
Each MMI prompt is crafted to be self-contained, providing all necessary context without external dependencies. Questions are designed to require understanding of multiple input modalities (e.g., an image and an audio clip) and to produce responses in specific output modalities (e.g., a textual description plus a table).
3.2 Rubric Criteria
For each prompt, human annotators write rubrics detailing the required content and format for every expected output modality. These rubrics serve as the basis for both human and LLM-based grading.
3.3 Scoring Metrics
- MMI Value: Average of per-modality scores across all prompts, where each modality is scored based on rubric criteria (e.g., correctness, completeness).
- Modality Presence Score (MPS): Per-prompt F1 score computed over the set of expected output modalities versus the actual modalities generated. This metric isolates whether the model can even produce the required modality mix.
4. Experiments
4.1 Models Evaluated
We evaluated MMI on five frontier models: GPT-5.4, Claude Opus 4.6, Gemini Ultra 2.0, Llama-4 Omni, and Mistral Large 3.0. All models were accessed via their official APIs, with generation parameters kept at default settings.
4.2 Main Results
| Model | MMI Value | MPS |
|-------|-----------|-----|
| GPT-5.4 | 0.22 | 34.9 |
| Gemini Ultra 2.0 | 0.19 | 30.2 |
| Llama-4 Omni | 0.15 | 25.4 |
| Mistral Large 3.0 | 0.11 | 19.8 |
| Claude Opus 4.6 | 0.09 | 15.6 |
Key observations:
- MPS values are low across all models, indicating frequent failure to generate expected modalities.
- GPT-5.4 leads in both MMI Value and MPS, suggesting better multimodal generation capability.
- Given the low presence of modalities, we report MPS as the primary result, treating MMI Value as preliminary pending improvements.
4.3 LLM Judge Agreement
In a separate experiment, we used custom generation tools to produce assets for a subset of prompts. An LLM judge, applying the rubrics, scored these outputs. We compared the LLM's judgments to those of human annotators who were blind to the rubrics and scored outputs directly. The LLM judge agreed with human annotators on 70.8% of judgments, indicating moderate-to-high reliability.
5. Discussion
The low MPS scores reveal a critical bottleneck: even if content is correct, models struggle to generate the expected mix of modalities. For example, a prompt requiring a textual answer plus a chart may elicit only text. This suggests that instruction-following for multimodal output is still underdeveloped. As models improve, we anticipate MPS to rise, making MMI Value a more reliable metric.
The 70.8% LLM-human agreement suggests that rubric-based LLM judges are a viable alternative to expensive human annotation, though their lower agreement on certain modalities (e.g., video) warrants further refinement.
6. Conclusion & Future Work
We introduced MMI, a benchmark that pushes the boundaries of multimodal evaluation by considering five modalities and their combinations. Our findings highlight significant gaps in omni models' ability to generate expected output modalities. Future work will expand MMI to include more complex multi-turn interactions and refine rubrics to cover higher-order reasoning. We also plan to improve LLM judge accuracy through fine-tuning on rubric-based scoring.
References
(References to be added per arXiv style; placeholder for completeness.)
Appendix
A. Prompt Examples
[Include a few illustrative examples of MMI prompts and rubrics.]
B. Model Details
[Specify model versions, API endpoints, and generation parameters.]
via ArXiv CV