Hour-Scale Timestamped Audio Captioning via Chunk-wise Processing
Jinchuan Tian, Sen Chen, Wenbin Huang, Chao Weng, Zhengwei Huang, Jiatong Shi, Yichen Lu, Haoran Wang, Shinji Watanabe, and Cong Zhou
Paper (arXiv, coming soon) · Project page · Quickstart · Model · Benchmark
LongCap describes what happens in a recording, from a short clip to hours of audio, as a timeline of events. This repository contains its inference code: a vLLM server and a client that you install with pip, plus a Docker image as an alternative.
- Timestamped audio captioning. LongCap locates the speech, music, and sound events in a recording, gives its start and end times, and describes it with structured attributes, such as the transcript and speaker traits for speech, the instruments and mood for music, or the source and setting for a sound, plus a short caption. Overlapping events are described separately.
- Speech recognition with speaker diarization. LongCap transcribes speech and labels who speaks when.
- Long recordings. LongCap works on short clips and on recordings an hour or longer, such as full films, shows, and meetings.
Attribute names and values are written in Chinese, while transcripts keep the spoken language; attribute_keys.json gives the English name of every key.
LongCap runs as a GPU server and a client. Install both in a Python environment, or use the Docker image for the server; the Quickstart covers both. With the server running, use either task:
# Timestamped audio captioning
longcap caption --audio recording.wav \
--model ./models/LongCap-35B-A3B-FP8 --output ./outputs/caption
# Speech recognition with speaker diarization
longcap diarize --audio recording.wav \
--model ./models/LongCap-35B-A3B-FP8 --output ./outputs/diarizationCaptioning writes timestamped events with their attributes to outputs/caption/predictions/predictions.jsonl; diarization writes timestamped segments with speaker labels and text to outputs/diarization/predictions/diarization.jsonl. The inference guide covers batches, resume, and output files, and the Docker guide covers running LongCap in a container.
| Release | Description |
|---|---|
| LongCap-35B-A3B-FP8 | Model weights in FP8; recommended, runs on a single GPU |
| LongCap-35B-A3B | Model weights in BF16; needs two GPUs |
| LongCapBench-TAC-Long | Human-annotated long-form benchmark |
LongCap is a research release, intended for research use. It is not a product or a hosted service and has not been validated for production use. Like other AI models, it can make mistakes: it may miss or invent events, give wrong timestamps, mistranscribe speech, or mislabel speakers. Attributes such as a speaker's gender, age, or emotion are inferred from audio and may be wrong or biased. Review outputs before relying on them, and do not use them to identify people or to make decisions about them. You are responsible for how you use the model and its outputs, including having the right to process the audio you provide. The software and weights are provided "as is", without warranty of any kind, under the Apache License 2.0.
@misc{tian2026longcap,
title = {LongCap: Hour-Scale Timestamped Audio Captioning via Chunk-wise Processing},
author = {Tian, Jinchuan and Chen, Sen and Huang, Wenbin and Weng, Chao and Huang, Zhengwei and Shi, Jiatong and Lu, Yichen and Wang, Haoran and Watanabe, Shinji and Zhou, Cong},
year = {2026},
howpublished = {Preprint},
url = {https://jctian98.github.io/longcap/}
}Jinchuan Tian and Shinji Watanabe, Carnegie Mellon University: jinchuat@andrew.cmu.edu, swatanab@andrew.cmu.edu. For bugs and usage questions, please open a GitHub issue.
We thank Xinying Hu, Shuangshuang He, and Zhuoqun Liu for their expert annotation work, and Anuttacon for its support of this work. The inference code is built on ESPnet and vLLM. The code, model weights, and benchmark are released under the Apache License 2.0; see third-party notices.