It has been about a month and I have fallen hard into my little image-captioning test project as a way to experiment with local agentic coding. And, I’ll admit, the actual application I have been working on as a side project has been fascinating as well. What I thought would be a simple app to caption images because I’m fairly lazy has turned into something much more sophisticated and fascinating.

Captioning Leads to Classification

I realized early on that if I want a useful caption, I first need to know what is in the image: who a person is, what type of animal or plant is present, how the image is composed, and so on. Throwing an image at a vision-language model (VLM) and asking it to describe the image is certainly cool and is about a million times better than what could happen 5 years ago, but I want more. I want real captions - like AP News worthy, and I want them grounded in truth - not hallucinated.

Vision-Language Model Strengths and Weaknesses

You can do some of that with vision-language models, but asking one model to handle everything at once leaves several important gaps. I tried the following models (local and hosted).

Local Models:

  • Gemma 4 26B: The best local model for general language tasks, but terrible at calculating bounding boxes
  • Qwen 3.5 9B: Not bad for 9B
  • Qwen 3.6 35B-A3B: This is by far my favorite local model for classification. Because it is a mixture-of-experts (MoE) model, it runs fast and has a relatively broad knowledge base.
  • Qwen 3.8 27B: Slow, with very precise bounding-box calculations and endless metadata, but not actually very knowledgeable

Hosted Models:

  • GPT Luna 5.6: This is probably what I will use in production when I am working with my own images.
  • Qwen 3.8 Flash Next: Not bad, but Luna just has better depth and doesn’t cost much more

But in the end, just trying to do it all with a VLM didn’t work. I needed much faster, more precise subject detection, so I went looking.

The Pursuit of More Deterministic Models

And I found ONNX, along with a broader world of detection and classification models, facial recognition, text and image embedding spaces, and more.

Local Subject Detection

Right now, the app can detect people, clothing, activities, faces, teams, wildlife, backgrounds, and more. It is about to put all of that information into a massive searchable database. Stay tuned to what Qwen 3.8 27B produces over the next few weeks!

OpenCode, Qwen 3.8 27B, and Work-Item Skills

While tuning this model to perform well with OpenCode, I found that the key is not fighting the model’s weirdness, such as its tendency to think endlessly. Instead, the workflow needs to be structured around it.

Opencode Learns Work Item Skills

In the end, I took a basic agentic workflow off the shelf and asked Qwen to create a set of skills for planning, implementing, reviewing, and retrospecting on each work item. Instead of just vibe coding, you can give this model a structured workflow, let it run all night, and come back to code that actually works. It is pretty amazing, but you have to add a substantial amount of structure or it will become frustratingly slow and tedious.

Until next time, let’s see what the world of local agentic coding brings!