Meric: A Unified Framework for Multimodal Music Generation and Retrieval via Representation Space Anchoring

Xihua Wang^*, Yinbo Wang^*, Jingchao Zhang^*, and Ruihua Song^†

Gaoling School of Artificial Intelligence, Renmin University of China

^* Equal contribution ^† Corresponding author

MERIC maps images, text, and video into a shared semantic embedding space, then uses that same space for both music generation and music retrieval.

ECCV 2026

Paper

Anchor Shared semantic embedding space as the music anchor

Training Stage 1 learns from 630K unpaired music tracks

Sampling One input can yield many faithful compositions

Unification The predicted anchor serves generation and retrieval

Overview

Cross-modal music generation from text and visual inputs is uniquely challenging due to its subjective, one-to-many nature. Current methods face three major limitations: they are bottlenecked by scarce paired data, lack explicit one-to-many modeling, and isolate generation from retrieval.

MERIC proposes a unified framework for multimodal music generation and retrieval. Its core idea is a semantic embedding space (a music semantic anchor), which decouples multimodal understanding from acoustic synthesis. The paper uses a two-stage training pipeline:

Stage 1: a Flow-DiT decoder learns to synthesize audio from music semantic embeddings using large unpaired music corpora.
Stage 2: the diffusion-based Music Head learns to map frozen Qwen3-VL embeddings into the music semantic space using paired multimodal-music data.

At inference time, the predicted anchor embedding is both a condition for acoustic synthesis and a query for music retrieval. This is the central reason generation, retrieval, one-to-many sampling, and the ablations below belong on one project page rather than separate demos.

Framework overview: multimodal queries (image, video, text) are encoded by a frozen Qwen3-VL-Embedding model. The Music Head maps query embeddings to a music semantic anchor, which serves as the interface for both generation (Flow-DiT Decoder) and retrieval (cosine similarity against pre-indexed music embeddings).

Paper Results

The demo samples below are meant to be heard next to the paper numbers, so the main generation and retrieval tables are surfaced here before the galleries. Lower FAD/KL is better; higher Sem and retrieval recall are better.

Vision Generation 0.127 FAD_C

MERIC on MelBench, below all listed generation baselines.

Text Generation 0.110 FAD_C

MERIC on MusicCaps, using the same anchor pipeline.

Retrieval One anchor, two tasks

The predicted semantic embedding is both a synthesis condition and a retrieval query.

Vision-to-Music Generation

Main baselines are evaluated on the full test set. Rows marked † are 2026 song-foundation cascade baselines evaluated on fixed N=500 subsets; their Sem scores are not directly comparable to full-test-set Sem.

Method	MelBench					MuImage					EmoMV					ARIA
Method	FAD_M	FAD_C	FAD_V	KL	Sem	FAD_M	FAD_C	FAD_V	KL	Sem	FAD_M	FAD_C	FAD_V	KL	Sem	FAD_M	FAD_C	FAD_V	KL	Sem
Art2Mus	31.6	0.945	13.7	0.0145	39.1	30.8	0.902	15.3	0.0151	45.3	33.9	1.109	14.6	0.0147	44.3	15.1	0.560	7.02	0.0077	45.3
GVMGen	3.55	0.483	5.78	0.0072	52.9	3.78	0.549	8.73	0.0086	49.6	3.96	0.617	7.20	0.0057	63.6	4.18	0.337	5.83	0.0052	60.0
AudioX	16.5	0.723	13.7	0.0127	53.8	13.9	0.623	14.4	0.0130	51.8	4.75	0.487	5.45	0.0071	79.0	20.1	0.766	11.7	0.0120	38.5
M2UGen	7.53	0.558	7.14	0.0090	55.3	5.19	0.421	8.66	0.0098	57.8	9.51	0.794	10.1	0.0087	54.6	24.0	0.744	4.89	0.0070	57.1
Qwen3VL + MusicGen	4.42	0.497	6.56	0.0076	80.6	2.99	0.418	8.93	0.0079	85.6	7.08	0.736	7.65	0.0083	80.3	5.09	0.283	2.04	0.0059	76.6
Qwen3VL + AudioLDM2-L	6.57	0.408	5.92	0.0078	77.6	4.59	0.274	6.08	0.0079	87.2	8.61	0.609	7.41	0.0086	78.7	9.95	0.262	3.66	0.0069	77.6
Ours	1.55	0.127	1.27	0.0059	74.7	1.79	0.169	3.20	0.0078	87.1	3.06	0.253	2.45	0.0060	81.0	3.30	0.247	2.08	0.0055	79.6
2026 song-foundation cascade baselines (N=500 subset) †
MERIC (N=500) †	1.96	0.156	—	—	80.2	2.24	0.183	—	—	88.1	3.32	0.272	—	—	73.7	9.23	0.496	—	—	65.2
YuE-7B †	12.83	0.518	—	—	61.5	8.93	0.384	—	—	69.8	15.21	0.710	—	—	68.6	19.25	0.657	—	—	64.4
HeartMuLa-3B †	11.35	0.534	—	—	51.2	10.64	0.705	—	—	48.2	14.61	0.711	—	—	53.0	16.06	0.819	—	—	60.2

Text-to-Music Generation

Method	MusicCaps			ARIA
Method	FAD_M	FAD_C	Sem	FAD_M	FAD_C	Sem
MusicGen-Medium	3.80	0.421	65.1	3.00	0.272	71.1
MusicGen-Large	3.51	0.409	67.1	2.55	0.248	72.4
AudioLDM2-Music	4.60	0.260	68.0	4.63	0.251	68.6
AudioLDM2-Large	4.17	0.178	76.7	3.93	0.211	77.5
Stable-Audio-Open	3.85	0.268	68.3	3.50	0.324	57.8
Ours	1.87	0.110	74.2	2.14	0.155	74.5

Vision-Music Retrieval (%)

MelBench: 5,000 one-to-one pairs. ARIA: 1,000 queries → ~9,000 audio candidates (one-to-many). Higher is better.

Method	MelBench				ARIA
	I→M		M→I		I→M		M→I
	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
ImageBind	4.38	18.00	3.66	17.00	0.30	2.30	0.13	1.77
Ours	1.34	8.62	1.12	9.54	0.40	2.80	0.39	2.40

Text-Music Retrieval (%)

CLAP is trained on 630K text-audio pairs. Ours routes text through Qwen3VLE → Music Head → anchor space without any text-audio training. Higher is better.

Method	MusicCaps				ARIA
	T→M		M→T		T→M		M→T
	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
CLAP	4.31	19.48	5.54	23.53	0.17	2.30	0.63	3.73
Ours	3.27	18.80	2.82	16.42	0.83	5.47	0.73	6.40

Unified Retrieval

MERIC does not add a separate retrieval model. The Music Head predicts an anchor in the shared semantic embedding space from a visual or text query; generation conditions the Flow-DiT decoder on that anchor, while retrieval ranks pre-indexed music embeddings by cosine similarity to the same anchor.

Image / Text / Video

Qwen3-VL Embedding

Semantic Anchor Space

Generate or Retrieve

Vision-to-Music Retrieval

For images and video frames, the same anchor used to drive synthesis can query an indexed music collection. This is why the demo keeps generation and comparison examples close to the retrieval table instead of treating them as separate systems.

ARIA I→M R@10: 2.80 vs ImageBind 2.30.

Text-to-Music Retrieval

Text follows the same route: text → Qwen3-VL → Music Head → semantic anchor. The paper result is notable because MERIC does this without direct text-audio training, unlike CLAP.

ARIA T→M R@10: 5.47 vs CLAP 2.30.

Image-to-Music Gallery

Curated visual inputs paired with music generated by MERIC. These examples make the anchor-based image-to-music behavior listenable before the controlled baseline comparisons.

One-to-Many Generation

The same visual input can inspire diverse yet equally valid compositions. The Music Head introduces controlled stochasticity: different random seeds yield different music semantic predictions, producing diverse audio outputs from a single image. Use Play All to hear each variation sequentially.

1 / 22

Text-to-Music Gallery

Text prompts are routed through the same Qwen3-VL → Music Head → semantic anchor path used by visual inputs. MERIC is not trained as a separate direct text-audio model.

Vision-to-Music Comparison

Compare audio generated by our method against baselines across multiple benchmarks. Click any play button to listen; only one track plays at a time. Use Play All to hear each method sequentially.

1 / 20

Text-to-Music Comparison

Compare MERIC against dedicated text-to-music baselines on MusicCaps. MERIC reaches these samples through the shared semantic-anchor route rather than direct text-audio training. Click any play button to listen; only one track plays at a time. Use Play All to hear each method sequentially.

1 / 10

Ablation Studies

Audio comparisons for key design choices. Click any player to listen; compare across conditions for the same input image.

Sampling Steps

The Music Head uses DDIM sampling to iteratively refine noise into a music semantic embedding. A single denoising step (s=1) produces collapsed, low-diversity embeddings; s≥5 are perceptually indistinguishable (p>0.2 in human MOS study), though s=20 best matches the ground-truth distribution.

Human MOS by DDIM steps — Human MOS ratings (n=100 images × 4 steps)

Cosine similarity distribution — Intra-group pairwise cosine similarity

Input image for MERIC sampling-steps ablation

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

s=1

s=5

s=10

s=20

Projector Architecture

Five projector architectures map frozen vision-encoder embeddings to the music anchor space. Linear (Contrastive) achieves the best retrieval but catastrophic generation quality (FAD 6.6× worse); Linear (MSE) produces good FAD but collapses retrieval by mapping to the conditional mean; VAE methods strike a moderate balance but share MSE’s deterministic collapse. Only the Music Head (diffusion-based) resolves this trade-off—best FAD with competitive retrieval—because iterative denoising produces predictions that are individually accurate yet collectively diverse.

PairCos vs FAD trade-off — Projectors in PairCos–FAD space: only the Music Head achieves both GT-level diversity and low FAD.

Input image for MERIC projector-architecture ablation

Ground Truth

Music Head

Linear (MSE)

Linear (Contrastive)

Bridge VAE

Cross VAE

Ground Truth

Music Head

Linear (MSE)

Linear (Contrastive)

Bridge VAE

Cross VAE

Ground Truth

Music Head

Linear (MSE)

Linear (Contrastive)

Bridge VAE

Cross VAE

Ground Truth

Music Head

Linear (MSE)

Linear (Contrastive)

Bridge VAE

Cross VAE

Ground Truth

Music Head

Linear (MSE)

Linear (Contrastive)

Bridge VAE

Cross VAE

Ground Truth

Music Head

Linear (MSE)

Linear (Contrastive)

Bridge VAE

Cross VAE

Ground Truth

Music Head

Linear (MSE)

Linear (Contrastive)

Bridge VAE

Cross VAE

Vision Encoder

Replacing CLIP ViT-H-14 (1024-d) with Qwen3-VL-Embedding (2048-d) consistently improves generation fidelity (FAD_MERT: 2.54 → 1.95 on MelBench), confirming that richer vision-language representations benefit cross-modal translation even with a frozen backbone. Both use the same Music Head projector.

CLIP ViT-H-14

Qwen3VL-Embedding

CLIP ViT-H-14

Qwen3VL-Embedding

CLIP ViT-H-14

Qwen3VL-Embedding

CLIP ViT-H-14

Qwen3VL-Embedding

CLIP ViT-H-14

Qwen3VL-Embedding

CLIP ViT-H-14

Qwen3VL-Embedding

Direct Conditioning

A direct Stage-2 baseline that conditions the decoder on Qwen3VL features without the shared semantic anchor trails MERIC on both FAD-MERT and semantic alignment. More importantly, it is an audio-only generator: it cannot produce a retrieval query in the music-anchor space, so it does not unify generation and retrieval.

Subset	Anchored FAD_M	Direct FAD_M	Anchored Sem	Direct Sem
ARIA	9.14	11.40	70.5	69.4
MuImage	2.23	2.23	87.7	83.6
MelBench	1.97	3.04	80.0	69.4

Noise Objective in Music Head

Diffusion and rectified flow are statistically indistinguishable when the anchor-space mapping is held fixed. This suggests the gains come from the anchor design rather than the specific generative formulation.

Subset	RDM FAD_C	FM FAD_C	RDM FAD_M	FM FAD_M
ARIA	0.496	0.494	9.23	9.14
MuImage	0.184	0.184	2.24	2.23
MelBench	0.156	0.155	1.96	1.97

Music-Anchor Encoder

Replacing the frozen music-semantic encoder shows that music-semantic pretraining is essential. CLAP, a general contrastive audio-text model, is an outlier even on its own aligned FAD-CLAP metric; The default music encoder and MERT perform similarly, confirming that the anchor design matters more than the specific encoder.

Metric	MERIC (default)	CLAP	MERT
FAD_CLAP (unbiased)	0.178	0.286	0.172
FAD_MERT (aligned)	1.31	2.35	1.01
Human Sem (N=50)	62.84	37.52	68.20

Citation

The official ECCV 2026 camera-ready paper is available below. Source code is Apache-2.0; checkpoints, pretrained components, and datasets follow their own upstream terms.

@inproceedings{wang2026meric,
  title     = {Meric: A Unified Framework for Multimodal Music Generation and Retrieval via Representation Space Anchoring},
  author    = {Xihua Wang and Yinbo Wang and Jingchao Zhang and Ruihua Song},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
  url       = {https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/2169.pdf}
}

Resources Paper GitHub Docs Model Weights Model Docs License Notes