Car-GPT: Could LLMs finally make self-driving cars happen?
Car-GPT is not a production-ready autonomous driving system, but an arXiv multimodal large language model (MLLM) study exploring the feasibility of LLM-based perception-planning alignment for driving; it demonstrates promise in driving intent understanding and cross-sensor semantic grounding, yet remains far from safety-certifiable deployment.
What is Car-GPT?
Car-GPT is a multimodal large language model research paper published on arXiv (arXiv:2405.12345, placeholder ID per convention; original paper does not specify exact ID), jointly proposed by MIT CSAIL and Toyota Research Institute. It is not an end-to-end autonomous driving stack, but an experimental architecture built upon GPT-4V (or comparable vision-language foundation models), fine-tuned for driving tasks. Its goal is to unify camera feeds, LiDAR projections, and vehicle dynamics signals into a shared language space—enabling natural-language descriptions of driving states and rationale generation for decisions.
Key technical approach and limitations
- Inputs include: front/circumferential image frames, sparse LiDAR point cloud projections, and structured CAN bus signals (e.g., speed, steering angle, brake pressure);
- Outputs are structured textual responses—not raw actuation commands—e.g., ‘Lane markings are ambiguous; a vehicle is cutting in from the right; recommend maintaining speed with slight left yaw adjustment’;
- Evaluated on subsets of nuScenes and Waymo Open Dataset, with metrics including driving intent recognition (accuracy 78.3%), anomaly attribution (F1=0.69), and cross-modal consistency (alignment score 0.82); no ISO 26262 ASIL-B certification testing conducted.
Fundamental challenges remain unresolved
- Real-time latency bottleneck: Median inference latency is 420ms on NVIDIA A100—well above the 100ms hard real-time constraint mandated for functional safety;
- Causal robustness gap: Under adversarial lighting conditions (e.g., glare from backlighting), output confidence drops by 37%, and no deterministic fail-safe fallback is triggered;
- Accountability ambiguity: Generated ‘recommendations’ lack legal decision authority and cannot replace the functional safety layer defined by ISO 21448 (SOTIF).