Frontier AI Models Tested in a Real-World Driving Experiment

A research team equipped a Toyota Corolla with an internet-connected laptop and a “comma-four” interface that links the car’s steering, throttle and brakes to a computer. Three large-language models—OpenAI’s GPT-6 Astra, xAI’s Grok 4.6 and Anthropic’s Claude Fable 5.1—were each given control of the vehicle in turn. The models received GPS telemetry and sensor data (steering-wheel angle, tire angle, etc.) and issued commands to the laptop, which relayed them to the car.

FuturismAI is changing how scientists discover materials and diagnose diseases

Only GPT-6 Astra succeeded in completing the entire course, doing so on its second attempt. The run covered less than 500 feet at a maximum speed of about 0.94 mph and lasted roughly five minutes. The other two models repeatedly failed to navigate the first corner, with errors traced to perception problems such as misreading lane boundaries and under-estimating the car’s width.

Futurism

Self-driving technology has long struggled with seemingly simple obstacles—unexpected stops on highways, collisions with barriers, and difficulty handling construction zones. The experiment was designed to test whether the newest frontier language models, which have shown dramatic performance gains on text tasks, could translate that capability into real-time vehicle control.

Futurism

• Course length: < 500 feet

Futurism

• Top speed: ~0.94 mph (? 1.5 km/h)

Futurism

• Duration: ~5 minutes per successful run

Futurism

• Compute cost: $7.74 for 6.6 million tokens processed by GPT-6 Astra

Futurism

• Fuel-cost comparison: The token expense is roughly 500 times the fuel cost for a car that gets 25 mpg at $4.60 per gallon, according to independent calculations.

Futurism

• Failure rate: The majority of attempts by Grok 4.6 and Claude Fable 5.1 did not pass the first corner, citing perception mismatches.

Futurism

The research team reported that the successful run demonstrates a “shocking and exciting” breakthrough, showing that frontier models can now operate a real vehicle at very low speeds. However, they emphasized that the result also underscores the need for extensive safety, alignment and evaluation work before any practical deployment. Some models, particularly GPT-6 Astra, occasionally refused to take control, citing built-in safety constraints even when the test environment was an empty lot with strict speed limits.

Futurism

The experiment provides concrete evidence that large-language models can interface with physical systems, but it also highlights the gap between raw model capability and the rigorous safety standards required for autonomous transportation. The high computational cost and the models’ propensity to halt operation for safety reasons suggest that additional engineering and regulatory frameworks will be essential before such systems can be trusted on public roads.

AI is changing how scientists discover materials and diagnose diseases

The team plans to expand testing to address the identified perception shortcomings and to develop more robust prompting strategies that keep the models engaged without triggering safety refusals. They also called for broader community efforts to create standardized benchmarks for evaluating AI-driven vehicle control, aiming to align future research with emerging legal and safety requirements.

Futurism