CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video–Language Models

A comprehensive evaluation framework that uses state-of-the-art text embedding models to measure the quality of video descriptions generated by vision-language models (VLMs). The framework computes semantic similarity between model predictions and reference descriptions at both document-level (coarse-grained) and sentence-level (fine-grained) granularity, and aggregates an ensemble of five embedding judges with Borda count.

Links: Dataset — GitHub · Hugging Face | Live leaderboard · HF mirror | Paper — arXiv:2608.04302 (EvalMG @ SIGIR 2026; see Citation)

CLIP-CC-Bench is the evaluation framework and leaderboard; CLIP-CC is the underlying dataset of 200 human-narrated movie-clip descriptions it scores against.

Overview

CLIP-CC-Bench is a text embedding-based evaluation framework that:

• Evaluates 17 vision-language models (VLMs) using an ensemble of 5 state-of-the-art text embedding models
• Ranks VLMs based on aggregated performance across all embedding model judges

Leaderboard

Result from the paper (frozen: V=17, Borda max 80) — the live leaderboard has the current pool (HF mirror). Borda = Borda count across the 5 embedding judges; Mean HM-CF = average harmonic mean of coarse- and fine-grained similarity across judges; full per-judge scores: results/embedding_models/aggregated_results/.

Rank	VLM	Borda	Mean HM-CF
1	VideoLLaMA3	80	0.67
2	mPLUG-Owl3	75	0.66
3	LLaVA-OneVision	67	0.64
4	ViLAMP	67	0.64
5	LongVU	61	0.63
6	Qwen2.5-72B	55	0.62
7	Qwen2.5-32B	48	0.61
8	VideoChat-Flash	42	0.60
9	MiniCPM-V	42	0.60
10	Video-XL	36	0.59
11	ShareGPT4Video	29	0.58
12	InternVL2	27	0.58
13	TimeChat	20	0.56
14	LLaVA-NeXT-Video	16	0.55
15	TS-LLaVA	9	0.53
16	Oryx	6	0.52
17	LongVA	0	0.48

Metrics are computed on 199 of the 200 clips (clip 126 is absent from the outputs of the 17 models evaluated in the paper).

Adding a model

The leaderboard takes post-publication submissions. The paper's 17-model result stays frozen in results/leaderboard/paper_v1.json; new models are ranked alongside it in the same pool.

1. Evaluate locally. Generate your model's predictions for the 199 evaluated clips in the data/models/{vlm_name}.json format (id → description), then run the released 5-judge pipeline (./run_all_evaluations.sh, or the per-judge scripts) to produce per-judge aggregated JSONs.
2. Open one pull request containing exactly:
   • your five per-judge JSONs at results/leaderboard/additions/<judge>/<slug>.json (identical schema to results/embedding_models/aggregated_results/<judge>/<vlm>.json);
   • one entry in results/leaderboard/models.json with display, params, params_b (total parameters in billions; null if undisclosed), added (YYYY-MM-DD), raw_outputs_url (REQUIRED — a publicly fetchable copy of your predictions JSON so maintainers can re-score), and optional hf_url;
   • the regenerated results/leaderboard/leaderboard.json (run python3 tools/build_leaderboard.py).
3. Do not edit README.md — its table is the frozen paper result and never changes with additions; the live leaderboard updates automatically after merge.
4. What CI checks: schema and counts (199 clips per judge), the frozen paper regression gate, that committed artifacts match the data (python3 tools/build_leaderboard.py --check), and registry hygiene (unknown keys refuse). A maintainer re-verifies provenance by re-scoring a spot check from your raw_outputs_url before merging. Predictions are NOT committed to data/models/ — that directory is the paper's frozen reproduction artifact.

Project Structure

CLIP-CC-Bench/
├── src/
│   ├── configs/                      # Per-judge evaluation configs (YAML)
│   │   └── requirements/             # Per-judge pip requirements
│   ├── scripts/                      # Per-judge evaluation scripts + rank_vlms.py
│   └── utils/                        # Model implementations and utilities
├── embedding_models/                 # Embedding-judge weights go here (not included)
├── venv_configs/                     # Per-judge virtual-environment setup scripts
├── data/
│   ├── ground_truth/                 # clip_cc_dataset.json (frozen CLIP-CC snapshot)
│   └── models/                       # The 17 paper models' predictions (frozen)
├── results/
│   ├── embedding_models/
│   │   └── aggregated_results/       # Canonical per-judge scores (<judge>/<vlm>.json + CSV)
│   └── leaderboard/
│       ├── paper_v1.json             # Frozen paper result (immutable)
│       ├── models.json               # Registry: display names, params, judges
│       ├── leaderboard.json          # Generated: the current ranking
│       └── additions/                # Community submissions (<judge>/<slug>.json)
├── web/                              # Authored page shell + Hugging Face Space card
├── tools/                            # Leaderboard generator + drift check + tests
├── .github/workflows/                # CI: PR validation + Pages/Space deploy
├── run_all_evaluations.sh            # Master evaluation script
├── LICENSE                           # MIT (code)
├── DATA_LICENSE                      # CC-BY-4.0 (reference summaries)
└── README.md

Running the pipeline regenerates the full results tree, including results/embedding_models/logs/, results/embedding_models/individual_results/ (per-video detail), and results/ranking/ (vlm_overall_ranking.csv, vlm_per_judge_metrics.csv). This repository ships the aggregated results behind the published numbers; the larger per-video artifacts regenerate on a run. dist/ — the deployable web artifact tools/build_leaderboard.py builds — is a build output and is gitignored.

web/ and .github/ are leaderboard-site and CI infrastructure — not needed to reproduce results. Slim clone: git clone --filter=blob:none https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench && cd CLIP-CC-Bench && git sparse-checkout set src data results tools venv_configs embedding_models.

Embedding Models

The framework uses an ensemble of 5 state-of-the-art text embedding models as judges:

Embedding Model	Parameters	Embedding Dim	Max Length	Implementation
NV-Embed-v2 (NVIDIA)	-	4,096	32,768	Custom local model
KaLM-Embedding-Gemma3-12B-2511 (PGFoundation)	12B	-	-	SentenceTransformer
llama-embed-nemotron-8b (NVIDIA)	8B	4,096	4,096	SentenceTransformer
gte-Qwen2-7B-instruct (Alibaba)	7B	3,584	32,768	SentenceTransformer
Qwen3-Embedding-8B (Qwen)	8B	Variable	32,768	SentenceTransformer

All models use official/recommended implementations from their respective HuggingFace model cards.

Setup

Prerequisites

• Python 3.9+
• CUDA-capable GPU (24GB+ VRAM recommended for large embedding models)
• ~70GB disk space for all embedding models
• PyTorch 2.0+
• transformers >= 4.42.0
• sentence-transformers >= 2.7.0

Installation

1. Clone the repository

git clone https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench.git
cd CLIP-CC-Bench

2. Setup virtual environments

Each embedding model requires an isolated virtual environment with specific dependencies. See venv_configs/README.md for detailed setup instructions:

# Setup all 5 virtual environments
# Follow instructions in venv_configs/README.md

Quick activation reference:

source venv_configs/activate_nv-embed_env.sh
source venv_configs/activate_kalm_env.sh
source venv_configs/activate_nemo_env.sh
source venv_configs/activate_gte_env.sh
source venv_configs/activate_qwen_env.sh

# Deactivate when done
deactivate

3. Download embedding models

Download the required embedding model weights from HuggingFace and place them directly in their respective directories under embedding_models/. Ensure model files are placed directly without creating subdirectories. Total storage required: ~70GB.

4. Prepare evaluation data

# Reference descriptions (ground truth)
data/ground_truth/clip_cc_dataset.json

# VLM-generated predictions
data/models/{vlm_name}.json

Reproducing Results

Option 1: Run All Evaluations (Recommended)

The master script runs all 5 embedding model evaluations sequentially and computes final VLM rankings:

./run_all_evaluations.sh

This script will:

1. Activate each embedding model's virtual environment
2. Run evaluation for all 17 VLMs using that embedding model
3. Generate individual results, aggregated statistics, and logs
4. Move to the next embedding model
5. Compute final VLM rankings across all embedding models
6. Save comprehensive results to results/ directory

Option 2: Run Individual Embedding Models

Evaluate using a specific embedding model:

# Activate the target environment
source venv_configs/activate_nv-embed_env.sh

# Run evaluation with config file
python src/scripts/run_nv_embed_evaluation.py --config src/configs/nv-embed.yaml

# Deactivate when done
deactivate

Repeat for the other embedding models (run_kalm_evaluation.py, run_nemo_evaluation.py, run_gte_evaluation.py, run_qwen_evaluation.py) with their respective configs and environments.

Option 3: Evaluate Specific VLMs Only

To evaluate a subset of VLMs (useful for testing):

source venv_configs/activate_nv-embed_env.sh

python src/scripts/run_nv_embed_evaluation.py \
  --config src/configs/nv-embed.yaml \
  --models internvl llava_one_vision longvu

deactivate

Generate VLM Rankings

After running all embedding model evaluations, generate the final rankings:

python src/scripts/rank_vlms.py

This creates results/ranking/vlm_overall_ranking.csv (final rankings) and results/ranking/vlm_per_judge_metrics.csv (per-embedding-model metrics).

Canonical results & data

The published numbers reported in the paper are summarized in the Leaderboard above; the machine-readable artifacts are:

• results/leaderboard/paper_v1.json — the frozen machine-readable paper result (immutable).
• results/leaderboard/leaderboard.json — the current machine-readable leaderboard (schema v2; carries source_fingerprint, per-judge scores, and provenance).
• results/embedding_models/aggregated_results/<judge>/<vlm>.json — the canonical full-precision per-judge results.
• results/embedding_models/aggregated_results/aggregated_results.csv — a 4-dp convenience derivative (the JSONs are canonical — the rank-8/9 tie is only resolvable at full precision).
• Live leaderboard: multimodal-intelligence-lab.github.io/CLIP-CC-Bench (HF mirror).

Data Format

Reference Descriptions (Ground Truth)

File: data/ground_truth/clip_cc_dataset.json — a frozen snapshot of the CLIP-CC dataset included here so the benchmark reproduces exactly.

[
  {
    "id": "001",
    "file_link": "https://www.youtube.com/watch?v=...",
    "summary": "The video begins with a man wearing sunglasses and a light-colored shirt driving a car..."
  }
]

VLM Predictions

File: data/models/{vlm_name}.json

{
  "001": "The video shows a man driving a car while appearing tense...",
  "002": "A person is shown lying down with headphones on, eyes closed..."
}

Community models' predictions live at the raw_outputs_url recorded in results/leaderboard/models.json; data/models/ stays frozen as the 17 paper models' predictions.

Citation

If you use CLIP-CC-Bench in your research, please cite:

@misc{ali2026clipccbench,
  title         = {{CLIP-CC-Bench}: Evaluating Paragraph-Level Video Descriptions in Video--Language Models},
  author        = {Ali, Mukhtiar and Dubey, Harsh and Mishra, Sugam and Pack, Chulwoo},
  year          = {2026},
  eprint        = {2608.04302},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  note          = {Presented at the 2nd Workshop on Evaluation for Multimodal Generation (EvalMG), ACM SIGIR 2026},
  url           = {https://arxiv.org/abs/2608.04302}
}

License

The evaluation code is released under the MIT License. The human-written reference summaries in data/ground_truth/ are released under CC-BY-4.0; the linked source videos remain the property of their original copyright holders (see DATA_LICENSE).

Acknowledgments

• NVIDIA for NV-Embed-v2 and llama-embed-nemotron-8b models
• PGFoundation for KaLM-Embedding-Gemma3-12B-2511 model
• Alibaba NLP for gte-Qwen2-7B-instruct model
• Qwen Team for Qwen3-Embedding-8B model
• HuggingFace for model hosting and the transformers library
• Sentence-Transformers for the embedding framework

Contact

For questions or issues, please open an issue on this repository, or contact the corresponding author: Chulwoo Pack (chulwoo.pack@sdstate.edu).