A local vision-language loop that drives a browser-based car simulator using screen capture and keyboard control.

The idea

This project explores whether a locally hosted vision-language model (VLM) can close the loop from pixels to physical actions — without cloud APIs, without a dedicated game SDK, and without any task-specific training data. The setup is deliberately minimal: capture the game window, send the frame to a VLM running on Ollama, parse a JSON action, and execute it as a macOS keystroke.

The target environment is City Car Driving running in a browser. Controls are limited to w (accelerate), a/d (steer), and s (brake). Each control loop iteration is: see → think → act.

Architecture

The agent follows a standard vision-language-action (VLA) pattern:

• Vision: Screen capture via mss, cropped to the game region and resized to reduce token load.
• Language / reasoning: Qwen2.5-VL (via Ollama) receives the image and a structured prompt, and returns a JSON action.
• Action: pyautogui translates the JSON into timed keypresses on the focused window.

I iterated through three agent variants, each building on the last:

• agent.py — The baseline. Full-resolution capture, default Qwen2.5-VL, single-key JSON output. Simple and readable, but slow (~10–16 s per inference).
• reasoning_agent.py — Switched to the 3B model variant, cropped out the sky, added chain-of-thought reasoning in the JSON response, and capped output tokens. Latency dropped to ~7–9 s per loop with more interpretable decisions.
• intelligent_agent.py — A hybrid approach. OpenCV runs lane-edge detection on each frame, overlays a red/green alignment guide and telemetry text onto the image, and passes that annotated frame to the VLM. The model gets both visual context and explicit lane-state text (e.g., "Drifting RIGHT → Recommend STEER LEFT"). This produced the most stable driving behavior.

Evaluating out-of-the-box VLMs

Before any fine-tuning, I wanted to understand what these models can do as-is on a continuous control task they were never trained for. The results were mixed but informative.

The full Qwen2.5-VL model could identify the road and generally chose to accelerate, but its ~15 s inference time made real-time control impossible — by the time an action executed, the scene had already changed. The 3B variant was fast enough to be usable (~7–9 s), and with chain-of-thought prompting it could reason about lane curvature and nearby barriers. However, it still struggled with sustained lane-keeping: it would drift, over-correct, and occasionally time out on longer reasoning chains.

The hybrid OpenCV + VLM agent was the most promising zero-shot configuration. By giving the model explicit lane telemetry rather than asking it to infer everything from raw pixels, steering corrections became more consistent. The model could focus on higher-level decisions (when to brake, when to steer hard) while the classical CV layer handled low-level alignment.

Key takeaways from the evaluation:

• Latency is the primary bottleneck — any loop above ~2 s makes closed-loop driving unstable.
• Prompt structure and output token limits matter as much as model size.
• Injecting structured telemetry (even simple OpenCV overlays) significantly improves decision quality without any training.
• Occasional timeouts and parse errors require fallback actions to keep the car moving.

Timing and model variants

I tracked end-to-end loop latency across configurations. Approximate numbers from my MacBook:

• Qwen2.5-VL (full) — 10–16 s inference. Too slow for closed-loop control.
• Qwen2.5-VL 3B — 7–9 s per loop with chain-of-thought. Usable but still sluggish.
• 3B + image crop + token cap — Cropping the top 30% (sky) and resizing to 320×240, combined with num_predict: 70, kept responses short and consistent.
• Hybrid OpenCV + 3B — ~8 s steady-state after an initial ~26 s cold-start loop. The first inference is always slower due to model loading.

Several engineering choices helped reduce latency: JPEG compression at quality 75, aggressive image downsizing, hard output token limits, and low temperature (0.1–0.2) for deterministic driving decisions. Action durations are clamped to 0.1–0.5 s bursts to prevent over-steering during slow inference gaps.

Tuning with reinforcement learning

The zero-shot agents can keep the car on the road for short stretches, but they lack the consistency needed for longer drives or complex intersections. The next step is to fine-tune the policy with reinforcement learning — using the VLM as a starting policy and improving it through reward signals from the game environment (staying centered, avoiding collisions, maintaining speed).

I have two early RL-tuned runs recorded below. These represent initial experiments where the agent learns from driving outcomes rather than relying purely on pre-trained VLM behavior.

RL run A — lane keeping with corrections

RL run B — extended driving session

What's next

• Reduce loop latency below 2 s — possibly via model distillation or a smaller specialized driving head.
• Formalize the RL reward function and training loop for sustained lane keeping.
• Test on more complex scenarios: intersections, traffic, and weather changes.
• Connect insights back to agricultural robotics, where similar VLA loops could drive field platforms from camera input.

Back to home