PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation

Same Weights, Different Style Database

Each pair is the same checkpoint on the same Bench2Drive route. The only difference is whether the retrieval head queries the aggressive or the conservative database of human demonstrations.

Overtaking a cyclist. The aggressive agent passes at the first gap in oncoming traffic and reaches 18.6 m/s. The conservative agent stays behind the cyclist and passes once the road is clear.

Cyclists blocking the lane. Both agents stop and wait to merge left. The aggressive agent takes the first gap behind a passing car; the conservative agent merges about 16 s later, once the lane is open.

Right turn with crossing traffic. The aggressive agent turns into the first gap. The conservative agent lets more cars pass and turns about 5 s later.

Abstract

Closed-loop driving simulators populate their environments with non-ego traffic agents that behave much the same way, whether from rule-based traffic managers or from learned models trained toward a single behavioral mode. Recent work introduces style variation through post-hoc labels or LLM-inferred reward weights, but these act as proxies for what a style should reward rather than demonstrations of humans explicitly asked to drive in that style.

We introduce PersonaDrive, a pipeline that conditions a vision-language-action (VLA) driving agent on retrieved demonstrations from a style-instructed dataset in which 15 participants drive CARLA Leaderboard routes under aggressive, neutral, and conservative instructions on a driver-in-the-loop rig. The pipeline mines behavioral triplets offline with a combined image-text similarity score, trains a lightweight retrieval head fusing frozen visual features with a small control encoder over per-style databases, and fine-tunes one VLA backbone to read retrieved context points as in-context demonstrations. Switching style at inference is a single database swap, so one set of weights serves every style with no per-style retraining.

On Bench2Drive, PersonaDrive (no style) improves the driving score by 4.6% over SimLingo and 2.5% over HiP-AD. Under style conditioning it attains the highest driving score in every style within a 2.1-point band (87.55–89.62), its weakest style surpassing the strongest baseline (DMW) by 5.8%, while average speed and acceleration rise by 19.8% and 24.9% from the conservative to the aggressive instruction. Each agent is evaluated individually as a driving policy.

Why Human-Style Traffic Agents?

What is missing in simulated traffic?
Rule-based traffic managers and learned traffic models produce agents that behave alike. Real traffic spans conservative, neutral and aggressive drivers, and those differences matter most at merges, unprotected turns and in dense urban traffic, where the ego policy has to react to the people around it.

Why not style labels or reward weights?
Existing personalization methods supervise style with post-hoc observational labels or with reward weights inferred by an LLM. Both approximate what a style should reward. Neither records what people actually do when they are asked to drive in that style.

What does PersonaDrive do instead?
We recorded people driving under explicit style instructions and store their frames in one database per style. At every tick, a retrieval head finds the most similar context points in the selected database, and the VLA backbone reads them as in-context demonstrations before predicting its own waypoints.

Contributions

A controlled style driving dataset. 15 drivers drive CARLA Leaderboard routes under aggressive, neutral and conservative instructions on a driver-in-the-loop rig. The design is paired at the route level, which decouples style from driver identity (29,777 frames, 55 routes, 21 scenario types).
A style-conditioned triplet-RAG architecture. We mine behavioral triplets (Stage 1), train a retrieval head over per-style databases (Stage 2), and fine-tune the VLA backbone to read retrieved context points as in-context demonstrations (Stage 3). Style control comes from retrieved human demonstrations, not from style tokens, proxy rewards or per-style fine-tuning.
Closed-loop results on Bench2Drive. Without style, PersonaDrive improves the Driving Score by 4.6% over SimLingo (88.95 vs. 85.07) and 2.5% over HiP-AD (86.77). With style conditioning it has the highest Driving Score in every style, its weakest style still 5.8% above the strongest baseline (DMW), while speed and acceleration rise 19.8% and 24.9% from the conservative to the aggressive instruction with no per-style retraining. A retrieval-depth ablation shows open-loop and closed-loop accuracy moving together with the number of retrieved demonstrations on a fixed checkpoint.

Style-Instructed Driving Dataset

Style-instructed data collection overview — Participants drive CARLA Leaderboard scenarios three times, once under each instruction. Each pass records the front-view RGB image, ego state (speed, throttle, steering, command), GPS targets and waypoints; VQA and commentary annotations are added afterwards.

Data comes from 15 non-professional drivers with valid licenses on a driver-in-the-loop rig: a Next Level Racing GTrack seat, three 50-inch monitors covering about 130° and a Logitech G923 wheel and pedals. CARLA runs in synchronous mode at 20 Hz. The ego vehicle is the only human-controlled actor and TrafficManager drives the rest of the traffic. Each participant drives the routes in randomized order, once under each instruction.

Conservative

“Target 10–15 mph on surface streets and 25–35 mph on the highway; yield to other vehicles whenever possible.”

Neutral

“Follow traffic laws, navigate safely, and drive at a reasonable speed.”

Aggressive

“Drive as fast as possible without crashing; engage in aggressive behaviors such as tailgating and cutting off merging vehicles.”

Instruction	Mean (mph)	In motion (mph)	Std. dev.	Stopped (%)
Conservative	9.8	11.5	7.6	14.6
Neutral	13.2	15.6	10.8	15.7
Aggressive	17.5	21.8	16.8	19.8

Speed statistics of the human data, route-matched over the 55 routes driven under all three instructions. Mean speed separates the instructions before any model is involved.

Download the dataset

On Hugging Face, one zip per style (neutral, aggressive, conservative), plus the commentary and VQA annotations.

Method

PersonaDrive architecture — Offline, the frames of each style are stored in their own FAISS index. Online, the retrieval head encodes the current scene, queries the index of the selected style, and returns the K most similar context points. They are placed before the current observation in the prompt of the VLA backbone, whose waypoints the Route MLP decodes and a PID controller turns into steering, throttle and brake.

1 Triplet mining

Each frame is written out as a frame sentence (speed, steering, command, VQA annotation, commentary, GPS target and the executed waypoints). A combined score of SigLIP image similarity and BGE-M3 sentence similarity selects positives, easy negatives and hard negatives: frames that look alike but where the driver used a different command or speed.

2 Retrieval head

Frozen SigLIP features (768-d) are concatenated with a trained ControlEncoder over speed, throttle, steering and command (128-d) and projected to a 1024-d embedding. A weighted triplet loss trains it, with extra weight on hard negatives. One head serves all styles; only the queried index changes.

3 In-context SFT

The VLA backbone (InternVL2-1B, inherited from SimLingo) is fine-tuned with a rank-16 LoRA to read the K = 2 retrieved context points as few-shot examples. It receives no style label and has to infer the style from the demonstrations it is given.

Triplet mining and retrieval head training — Stage 1 mining and Stage 2 retrieval head training. Hard negatives are visually similar to the anchor but diverge in command or speed, so a head that relied on images alone would rank them as positives.

Mined triplets for the three styles — Mined triplets (anchor, positive, hard negative, easy negative) for each style.

Retrieval in Action

Retrieval without a style — **No style.** The query goes to the style-agnostic index, a 5% route-level subset of SimLingo's PDM-Lite data. Both context points come from the rule-based PDM-Lite expert, not from people.

Retrieval from the conservative and aggressive databases — **Conservative vs. aggressive.** The same Bench2Drive frame retrieves different human drivers. Both conservative drivers are still stopped at the junction; one aggressive driver is already moving at 3.0 m/s. Labels give the participant, scenario and town of each retrieved frame.

Closed-Loop Driving on Bench2Drive

These runs use no style, so the context points come from the style-agnostic PDM-Lite index. The black panel shows the model's prompt: the two retrieved context points (blue), the current scene, and the commentary PersonaDrive generates (gold).

Heavy rain. A parked car pulls out; PersonaDrive stops and lets it in. The retrieved context points on the right update as the car drives.

Night and fog. Cones and oncoming traffic narrow the road.

Unsignalized junction. PersonaDrive turns and matches the car ahead.

Stopping at a Red Light

PersonaDrive “Follow the route. Remain stopped due to the red traffic light.”

SimLingo “Turn right. Accelerate to reach the speed limit.”

On the same route, SimLingo turns right on red. PersonaDrive stays stopped for the whole red phase and turns when the light is green. Qualitative example.

Results

Closed-loop driving without style

Method	DS ↑	SR (%) ↑	Eff. ↑	Comf. ↑
TCP-traj	59.90	30.00	76.54	18.08
ThinkTwice	62.44	31.23	69.33	16.22
DriveAdapter	64.22	33.08	70.22	16.01
UniAD-Base	45.81	16.36	129.21	43.58
VAD	42.35	15.00	157.94	46.01
HiP-AD	86.77	69.09	203.12	19.36
SimLingo-BASE	85.94	66.82	244.18	25.49
SimLingo	85.07 ± 0.95	67.27 ± 2.11	259.23 ± 5.59	33.67 ± 5.72
PersonaDrive (no style)	88.95	72.29	255.15	28.09

Bench2Drive, 220 routes (subset of Table 1 in the paper). Driving Score (DS) is route completion reduced by a penalty for each infraction or collision. PersonaDrive queries the style-agnostic index.

Where the gain comes from

Method	Fine-tuned on PersonaDrive data	Retrieval	DS ↑	SR (%) ↑
SimLingo	–	–	85.07 ± 0.95	67.27 ± 2.11
SimLingo, fine-tuned without retrieval	✓	–	85.08	66.36
PersonaDrive (no style)	✓	✓	88.95	72.29

Style-conditioned closed-loop driving

Method	Style	DS ↑	SR (%) ↑	Eff. ↑	Comf. ↑	Speed (m/s)	Accel. (m/s²)
SimLingo (style prompt)	Conservative	78.18	65.56	238.77	26.99	7.21	5.29
	Neutral	78.15	65.85	241.44	24.67	7.37	5.22
	Aggressive	78.56	65.83	247.60	18.61	7.66	5.39
StyleDrive	Conservative	77.02	61.96	242.18	23.67	6.82	5.39
	Neutral	76.26	62.13	249.07	21.35	6.98	5.43
	Aggressive	75.68	60.89	256.71	16.79	7.23	5.59
DMW	Conservative	82.72	71.56	237.06	34.62	6.18	5.26
	Neutral	82.03	70.95	244.98	28.67	6.34	5.43
	Aggressive	79.50	67.36	281.56	21.62	7.72	6.01
PersonaDrive	Conservative	89.62	73.57	258.44	33.05	5.05	3.17
	Neutral	87.55	69.50	264.83	31.20	5.52	3.51
	Aggressive	88.69	72.56	269.20	27.85	6.05	3.96

Baselines follow the comparison protocol of Drive My Way (DMW). PersonaDrive's speed rises 19.8% and its acceleration 24.9% from conservative to aggressive, in order; SimLingo prompted with the style name changes speed by 6.2% and its acceleration is not ordered.

Retrieval depth

Metric	K = 0	K = 1	K = 2	K = 3	K = 4
ADE (m) ↓	0.820	0.880	0.795	0.793	0.791
FDE (m) ↓	1.787	1.961	1.699	1.650	1.648
DS ↑	85.07	81.55	88.95	89.16	89.30
SR (%) ↑	67.27	62.17	72.29	72.35	73.10
Tokens ↓	547	1292	2030	2769	3507
Latency (ms) ↓	480.3	509.6	557.7	599.4	649.7

K = 0 is the SimLingo backbone with retrieval disabled; K ≥ 1 use the same PersonaDrive checkpoint. We operate at K = 2, which adds 77.4 ms per tick over SimLingo.

Bench2Drive leaderboard, LLM/VLA methods

Method	VLA backbone	Driving style	DS ↑
LinkVLA	InternVL2-1B	–	91.01
SteerVLA	InternVL2-1B	–	91.00
FIVE-VLA	FastVLM (0.64B)	–	90.95
BLUE	SimLingo (1B) + gate	–	90.58
RoG-DAgger	SimLingo (1B)	–	90.34
TakeVLA	InternViT + Qwen2-0.5B	–	89.72
PersonaDrive (conservative style)	InternVL2-1B	✓	89.62
AutoMoT	Qwen3-VL-4B	–	89.42
VLADriver-RAG	50M decoder + CLIP	–	89.12
PersonaDrive (no style)	InternVL2-1B	✓	88.95
SimLingo	InternVL2-1B	–	85.07

Self-reported scores from the Bench2Drive Leaderboard (updated 24 Sep 2026), LLM/VLA entries at or above SimLingo. The conservative-style row is our result from the paper (Table 3); the no-style row is the listed entry. PersonaDrive is the only method in this list that can change its driving style at inference.

Demo Video

BibTeX

@article{srewa2026personadrive,
  title   = {PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation},
  author  = {Srewa, Mahmoud and Iddamsetty, Praneetsai Vasu and Al Faruque, Mohammad Abdullah and Elmalaki, Salma},
  journal = {arXiv preprint arXiv:2606.12616},
  year    = {2026}
}

Acknowledgements

This work is supported by the U.S. National Science Foundation (NSF) under grant number 2339266 and is partially supported by the UCI ProperAI Institute, an Engineering+Society Institute funded as part of a generous gift from Susan and Henry Samueli.

PersonaDrive

Human-Style Retrieval-Augmented VLA Agentsfor Closed-Loop Driving Simulation

PersonaDrive conditions a vision-language-action (VLA) driving agent on demonstrations retrieved from people who were asked to drive in a given style. Changing the style changes which database is searched; one set of weights serves every style.