CoVLM-Drive
A unified VLM baseline: the ego image, the roadside image, and the prompt enter a single input sequence; the backbone is adapted with LoRA; and structured command and waypoint heads read the pooled prompt representation.
A Real-World Benchmark for Cooperative Driving
Question Answering and Planning
Background: a validation sequence from V2X-Seq-SPD, shown unmodified.
Vision–language models have made substantial progress in autonomous driving, but their success has primarily been studied in ego-centric scenes. Infrastructure-side observations provide views beyond the ego vehicle's field of view, yet conventional cooperative-driving systems typically transform them into geometric representations for downstream perception and planning. Directly incorporating these views into VLMs offers an opportunity to improve cooperative scene understanding and trajectory planning. However, question answering and trajectory planning have not been jointly evaluated on the same real-world vehicle–infrastructure scenes.
We present CoVLM-Bench, a benchmark for cooperative driving question answering (CDQA) and cooperative planning (CP) on vehicle–infrastructure paired scenes. It provides scene-grounded CDQA annotations, three-part rationales as auxiliary supervision, and future trajectory targets derived from recorded ego motion. It contains 2,196 paired frames with 35,136 CDQA annotations, while CP predicts six waypoints over a three-second horizon. The annotations combine model-assisted drafting, record-based computation, and human verification. Built upon CoVLM-Bench, we introduce CoVLM-Drive, a unified VLM baseline that directly uses paired views for both tasks. Experiments show that CDQA adaptation improves answer accuracy and that CoVLM-Drive reaches a lower FDE than the compared V2X planners; QA initialization and rationale supervision each reduce planning error.
Both tasks are annotated on the same paired vehicle- and infrastructure-side frames, so understanding and planning are measured on identical scenes.
Scene-grounded questions over traffic participants, ego maneuvers, and roadside observations, answered from the paired views and a neutral object catalog.
Six waypoints over a three-second horizon, with targets derived from recorded ego motion, scored by L2 at each horizon and by final displacement error.
Scene overview, V2X-aware critical objects, and decision reasoning, provided as additional supervision rather than as a separate task.
Consecutive validation frames at 0.5 s intervals. Each frame shows the ego and roadside views, a bird's-eye view with the recorded trajectory and the planners' trajectories, and the command and final displacement error of each planner. Every value shown is read from the evaluation artifacts. UniV2X's raw command ids are not mapped onto the benchmark command vocabulary, so no command is shown for it, matching the dashes in the table below. These three sequences are selected examples, not an average; the aggregate over all 654 validation examples is in the table.
Sequence 0085 · 16.5 s, a right turn. On average 9.6 objects per frame are annotated only in the roadside view. Mean FDE over the clip: CoVLM-Drive 3.79 m, UniMM-V2X 5.60 m, UniV2X 5.61 m.
A unified VLM baseline: the ego image, the roadside image, and the prompt enter a single input sequence; the backbone is adapted with LoRA; and structured command and waypoint heads read the pooled prompt representation.
654 validation examples. L2 is the mean distance to the reference waypoints at each horizon and FDE is L2 at 3.0 s. Efficiency uses batch size one on an RTX 4090.
| Method | 1.0 s | 2.0 s | FDE ↓ | Cmd Acc. ↑ | Lat. (ms) ↓ | FPS ↑ |
|---|---|---|---|---|---|---|
| Task-trained V2X planners | ||||||
| UniV2X | 2.079 | 3.831 | 5.861 | – | 812.7 | 1.23 |
| UniMM-V2X† | 1.677 | 3.427 | 5.357 | 0.8165 | 523.9 | 1.91 |
| MAP† | 1.408 | 3.233 | 5.240 | 0.8394 | 588.6 | 1.70 |
| CoVLM-Drive backbone variants | ||||||
| Qwen3-VL-8B · standard CP | 1.500 | 3.119 | 4.911 | 0.7706 | 223.1 | 4.48 |
| Qwen3-VL-8B + ego status | 1.515 | 3.207 | 5.126 | 0.7401 | 180.1 | 5.55 |
| Qwen2.5-VL-7B | 1.530 | 3.241 | 5.120 | 0.7569 | 273.8 | 3.65 |
| InternVL3-8B | 1.493 | 3.145 | 5.076 | 0.7324 | 305.1 | 3.28 |
| LLaVA-OneVision-7B | 1.555 | 3.284 | 5.272 | 0.7385 | 280.9 | 3.56 |
| Idefics3-8B | 1.691 | 3.525 | 5.538 | 0.7492 | 1070.6 | 0.93 |
| Zero-shot VLMs | ||||||
| Qwen3-VL-8B | 3.640 | 7.311 | 11.227 | 0.6468 | – | – |
| InternVL3-8B | 6.828 | 13.397 | 20.007 | 0.6774 | – | – |
† our protocol-aligned reproduction; UniV2X is evaluated from the released predictions. The standard configuration reaches a lower FDE than the compared V2X planners and runs with lower latency, while MAP retains the stronger command metrics. Trajectory accuracy against recorded ego motion is an open-loop measurement, not a closed-loop safety evaluation. The full table is in the repository.
Predicted trajectories compared with the recorded ego motion.
The repository is released in stages. Items are listed in intended order; no dates are promised.
@misc{covlmbench,
title = {CoVLM-Bench: A Real-World Benchmark for Cooperative Driving
Question Answering and Planning},
note = {Preprint coming soon},
year = {2026}
}