Part 3: Re‑Architecting the Autonomous‑Driving Stack with Tool‑Using LLMs

Autonomous vehicles (AVs) confront ultra‑diverse road scenes, harsh latency budgets and unforgiving regulators. Post‑Transformer tech is letting teams rebuild the AV brain around modular, verifiable LLMs.

AV Challenge	Technique	2025 Frameworks
Fusing camera + LiDAR + maps under uncertainty	Multimodal unification + Multi‑agent orchestration	DriveAgent spawns vision, LiDAR and causal‑reasoning agents for consensus driving plans.
Scaling to global skill sets (snow, dense urban, highways)	Scene‑ and Skill‑specialised MoE	DriveMoE achieves SOTA on Bench2Drive closed‑loop tests.
5–10 s horizon prediction without labels	Latent World Models (LAW)self‑supervise future latent features.
	Edge deployment on DRIVE Thor	SSM + INT4 + 2:4 sparsity supported in DriveOS LLM SDK.
Regulatory traceability	Runtime guardrail DSLs enforce no‑go zones (e.g., solid lines, speed caps).	Draft IEC 61508 annex proposal.
Hallucination‑free routing	RAG 2.0 inside forward pass queries HD maps mid‑inference.	Baidu Apollo RAG‑inside branch (closed beta).

Key shift: From monolithic perception‑→‑planning pipelines to tool‑using LLM agents that can cite sources (maps, sensor logs), prove invariants on the fly, and squeeze onto automotive SoCs.