Status

This project is in the early planning stage. The goal is to extend the vision-language-action driving loop I built for City Car Driving to Grand Theft Auto: San Andreas — a full 3D open-world environment with richer visuals, more complex traffic, and varied terrain.

Why San Andreas?

The browser-based City Car Driving simulator was a good sandbox for prototyping the VLA loop, but it is limited: fixed camera angle, simple graphics, and predictable road layouts. GTA San Andreas offers a much harder and more realistic testbed — dynamic NPC traffic, intersections, hills, weather, and a third-person camera that moves with the vehicle. If a VLA agent can drive reasonably in San Andreas, it is a stronger signal that the approach generalizes.

Current blockers

• Game access. Running GTA San Andreas on my Mac requires purchasing the game (the paid version through Steam or Rockstar). I do not yet have a licensed copy set up on my current machine.
• Platform compatibility. San Andreas is a Windows-native title. Running it on macOS means either Boot Camp (not available on Apple Silicon), a Windows VM, or a compatibility layer like CrossOver/Wine — each with performance trade-offs that affect real-time screen capture and control.
• Screen capture region. Unlike the browser game where the window position is fixed, a full-screen or windowed GTA instance will need dynamic region detection or a consistent windowed layout for reliable frame capture.
• Control mapping. GTA uses different key bindings and physics (handbrake, camera rotation, weapon switching) that the current agent does not handle. The action space will need to be redesigned for the 3D environment.

What I am planning

• Purchase and install the licensed game on a Windows setup (either a dedicated machine or VM) for stable performance.
• Adapt the existing VLA loop (capture → infer → act) with a windowed capture region and GTA-specific control prompts.
• Start with simple tasks: drive straight on a highway, follow a road without crashing — before attempting city navigation or mission objectives.
• Explore whether the OpenCV lane-assist overlay from the City Car Driving hybrid agent transfers to GTA's third-person camera, or whether a different telemetry injection strategy is needed.
• Eventually connect RL fine-tuning, similar to the City Car Driving experiments, to improve sustained driving performance.

I will update this page once the environment is set up and the first agent runs are recorded.

Back to home