Visually Prompting Moondream
May 9, 2026
Try the demo on Hugging Face →
Motivation
I loved Legos as a kid, and recently started building with them again. The main difference between then and now is I have been trained as a consultant to find any process inefficiencies, and think of ways to improve them. While building my newest set, my inner consultant quickly overwhelmed the inner child in me. To build, the setup is simple. You have two things in front of you. The instruction manual, and a pile of Legos. The manual tells you what you need to grab, and where to put it. The slow part is searching through the pile to find that piece. What if you could do this faster? I envisioned an application where you can show a model an image of the candidate piece, and it will return it's exact location in the pile.
While this kind of defeats the purpose of building Legos in the first place, it's a great motivating example for the bigger problem: sometimes you want to find something in an image and you can't easily describe it in text. You just have a picture of it.
Idea: use Moondream
Moondream AI (and Moondream 2 in particular) is a small ~2B parameter open-weights vision-language model. It already knows how to encode images, and it already has a built-in detect function that returns bounding boxes for an object provided as text — e.g. baseball hat or red mug.
So the question that took over my weekend was: it can encode images, and it can already detect things from text. Can we make it detect things from another image?
Approach
As a motivating example of what we want, here's the kind of result we want. Given a query image of an object and a target image, find every instance of that object in the target. Below is an example from the trained model:
Images from the LVIS dataset.
That's the behavior we're after. The plan is to keep Moondream's detect service as-is, but swap out the text class name for image tokens from a query image. Image tokens and text tokens go into the model the same way, so in theory this should just... work.
A couple of questions had to get answered before fine-tuning:
- Do we need to consolidate the 729 image tokens that come out of the vision encoder, or can we splice them in directly? (I went with mean-pooling them into a single token to start, just to keep the prompt shape similar to the text version.)
- Does the embedding space of these query tokens already live somewhere relatively close to where the text token for that object would live? In other words, does Moondream "see" a fork image and the tokens for the word "fork" the same way out of the box, even before any training?
Spot-checking a few examples, the answer to the second question was a soft yes. Close enough to be worth fine-tuning. For more details, see this notebook.
For training, I attached LoRA adapters to the vision projection MLP (the small piece that maps ViT features into the text decoder's space) and fully fine-tuned the region prediction head. In theory you could go even more modular and train only the vision projection, or wrap the query encoder in a small dedicated model, but this combo was a reasonable starting point.
How is fine-tuning done?
For data, I used LVIS, a dataset of everyday objects with lots of examples of the same class across many images. Perfect for quickly assembling triplets without building anything custom.
Each training example is a triplet:
- Query image: the bounding-box crop of an object from one sample image (e.g. a fork crop).
- Target image: a different image that contains at least one instance of the same class.
- Boxes: all the ground-truth bounding boxes of that class in the target image.
So if you have a fork crop pulled from one photo, you prompt the model with that crop and a different photo, and hope it draws bounding boxes over the forks in the target.
For the loss itself, I used a "teacher-forcing" approach that worked well for me in the past. Moondream predicts discrete bins for x-center, y-center, width, and height of each box. So for every ground-truth box in the target image, you get four cross-entropy terms (one per axis) against those bins. After each axis is predicted, the ground-truth value gets fed back in as the next input — that's the "teacher forcing" part, and it lets the model learn each axis without error compounding across the box, which in my experience was causing too much instability in training. The full loss is just the average of those four cross-entropy terms across all boxes in the sample (capped at max_objects_per_sample to keep things bounded).
The transfer is not perfect. A fork from one kitchen doesn't look identical to a fork from another, lighting and angles differ, etc. So there's a real ceiling to how well this dataset can train the model. But it's enough to see whether the idea works at all and whether it's worth more attention I've given it so far.
Results
Results agree with the premise. The F1 score on a held-out set of classes not used during training goes from roughly 0.09419 at the start of training to around 0.34195 after fine-tuning! Not amazing in absolute terms, but a clear improvement (over 3x) that says the model is learning to read the visual prompt.
Here are a couple of examples from the validation set. Each figure shows the query crop on the left, and the target image on the right with ground-truth boxes in green and the model's predictions in magenta:
And a representative failure mode:
slide. The class here is "slide," but the dominant subject of the query crop is the child in front of it. Mean-pooling smears the kid and the slide into a single token, so the model ends up boxing the person in the target image instead of the slide — a clean illustration of why a single pooled vector is a lossy way to represent a query, and why a more-targeted pooling (or attentive pooling conditioned on the target) is the obvious next step.I ran out of GPU credits on Lightning before I could finish a proper grid search, but the runs and code so far are available here and here.
Out-of-distribution test
To reiterate, the fine-tune above was trained entirely on LVIS , common everyday objects (forks, pandas, slides, mugs). So I wanted to see what would happen if I gave it a query crop of a single basketball player see what it would do:
This is a a pretty promising sign that the model can use query images as a way to guide object detection. This result, and any others you would like to try, can be tested on the Hugging Face demo.
Excited for any feedback, ideas for better pooling / training setups, or suggestions for what else to build on top of Moondream next!
Related work
First, the obvious one: this whole experiment is built directly on top of Moondream. Without its pre-trained detect capability and small footprint, none of this would have been tractable on a weekend budget.
The broader idea of using an image as a detection prompt is not new. A non-exhaustive list of prior work that does some flavor of this idea: or segmentation:
- T-Rex and T-Rex2 : interactive object detection driven by visual exemplars (boxes/points), with T-Rex2 unifying text and visual prompts in one model.
- DINOv : visual in-context prompting for generic segmentation and detection.
- OWL-ViT (and OWLv2) : open-vocabulary detection that also supports image-conditioned one-shot detection by swapping the text query embedding for an image embedding.
- CountGD : extends GroundingDINO with visual exemplars for open-world counting.
- SAM : not detection per se, but the canonical example of a model designed from day one around visual prompts (points, boxes, masks).
So visual-prompted detection itself is well-trodden ground. What I haven't seen written up and what felt worth posting about is doing this as a transfer from a text-conditioned detector inside a general-purpose VLM.