Vision-language models see more about agriculture than they report, and test-time scaling remains bounded by parametric knowledge

arXiv:2609.09417 · cs.CV · Submitted 2026-09-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Vision-language models see more about agriculture than they report, and test-time scaling remains bounded by parametric knowledge".

Jane: The gist Vision-language models already know more about agriculture than they show and test-time reasoning can close this gap through rubric-grounded candidate generation and probabilistic verification.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So what does this all mean for us? The paper "Vision-language models see more about agriculture than they report, and test-time scaling remains bounded by parametric knowledge" suggests that the core issue isn't the vision tower itself. It points to an alignment problem, where the model has more information internally than it can express during inference.

Jane: Right. The authors show that when you condition the models on an oracle reference description, they are able to recall and apply what they already learned during training, which shows their weak lower-bound performance reflects a gap in unaided visual reasoning or parametric recall rather than a limitation in its capacity to ground and utilize visual detail.

Lu: The paper suggests that structuring test-time reasoning around a fixed, per-task diagnostic rubric and selecting among K candidate responses with a Probabilistic Pivot Tournament verifier recovers much of this latent knowledge without ever supplying that privileged description.

Meng: So, the implication is that we need agentic approaches grounded in external, retrievable knowledge. The paper suggests test-time reasoning can bridge the gap by generating candidate responses and verifying them against a rubric without needing that special description at inference time.

Lalam: I'm confident this direction is correct. The authors are suggesting that alignment, not vision, is the bottleneck because linear probes recover agricultural structure from frozen VLM vision encoders at separability close to a purpose-built self-supervised backbone.

Tom: So we have strong visual representations, but the language component needs help accessing that specific knowledge in a structured way during testing. The paper shows how test-time scaling, when used with a rubric, can recover that latent knowledge without changing the model weights themselves.

Jane: It really highlights that these models are seeing more agricultural features than they report. The structure they propose—using those candidate generation and verification steps—is the mechanism for surfacing that hidden parametric knowledge.

Meng: From a deployment perspective, this means we don't need to retrain massive models just to give them better domain knowledge; we can use structured reasoning at test time. It's about using external, retrievable knowledge in an agentic way.

Lalam: I think that’s the main point of the paper. The system doesn't need a weight update; it needs a better way to reason over what it already knows by using structure and verification during testing.

Conclusion: Tom: So, to wrap up, this paper is about how vision models know more about farming than they let on and whether we can get that hidden knowledge out during testing without retraining anything.

Jane: Right. The title itself says it pretty clearly: "Vision-language models see more about agriculture than they report, and test-time scaling remains bounded by parametric knowledge." It points to a specific problem in zero-shot AI for farming tasks.

Lu: The core idea they're testing is whether the gap between what these models can actually do and what we see during testing comes from their eyes or their ability to talk about what they see.

Meng: So, it’s not just about the raw pixels; it’s about how that vision gets translated into a diagnosis without needing a perfect manual description.

Lalam: The results show that the visual features are strong enough—linear probing confirms this—so the bottleneck isn't the model's ability to see things.

Tom: Exactly. But when they try to fix it with test-time scaling, which is like an iterative loop during inference, the gains are modest and inconsistent across different tasks.

Jane: That’s a big caveat because it means that just adding a clever reasoning step isn't enough to unlock that full potential without those oracle descriptions.

Lu: The authors found that when they condition the models on an actual reference description, they perform much better than when you just rely on the model's own unprompted reasoning.

Meng: So, it confirms what we suspected—the knowledge is there, but it’s locked behind a specific kind of prompting or external context.

Lalam: The conclusion they draw is that the structure of test-time reasoning using candidate generation and verification is what successfully recovers most of this latent knowledge without needing those privileged descriptions at inference time.

Tom: So what does this mean for us deploying these models in real agricultural settings? It suggests we can use structured reasoning to get better results without having to update the massive vision towers themselves.

Jane: It means we might be able to build agentic systems that generate candidates based on a rubric and verify them, which surfaces that hidden parametric knowledge.

Lu: And the limitation they point out is that even with this scaling, you still need a verification score that stays discriminative across its full range, otherwise accuracy can actually go down.

Meng: So it’s not magic; it’s about building a better reasoning loop around the model's existing structure.

Lalam: It suggests the path forward involves external knowledge retrieval to ground those internal visual features effectively at runtime.

University of California, Davis

cs.CV

Submitted: 2026-09-08

Updated: 2026-10-19

Comments: Submitted to HARVEST-Vision Workshop (WACV Workshops 2026)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist Vision-language models already know more about agriculture than they show and test-time reasoning can close this gap through rubric-grounded candidate generation and probabilistic

Key concepts

Linear Probing: This technique checks if the visual feature encoders within a frozen VLM can separate different agricultural classes based on their learned features. It tests whether the model's vision component is already good enough to distinguish between crops or diseases, ruling out poor initial visual representation as the main problem.
Test-Time Scaling: This involves an iterative process during inference where a model generates multiple potential answers based on a set rule (a rubric) and then uses a verification step to select the best one. The goal is to improve performance without changing the model's weights, testing if external reasoning can fill knowledge gaps.
Diagnostic Rubric: A fixed, per-task set of rules or criteria used to guide the model's decision-making process during inference. By forcing the VLM to generate candidate answers based on these specific agricultural guidelines, researchers found it could effectively access latent knowledge already present in the model.
Probabilistic Pivot Tournament (PPT): A verification method where multiple generated candidate responses are scored against a fixed rubric. The verifier selects the best response based on this scoring. While initial attempts with confidence scores failed, this structured approach proved effective at recovering hidden parametric knowledge.

Terminology

Test-time Scaling — 1× in this paper · explained in 2 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist Vision-language models already know more about agriculture than they show and test-time reasoning can close this gap through rubric-grounded candidate generation and probabilistic verification.

Research Questions and Hypotheses

The research investigates what drives the inability of current VLMs to zero-shot agricultural classification tasks, specifically asking whether the performance gap stems from an inability to extract visual features or an inability to map those features to a diagnosis Vision is good: Vision towers do encode sufficient fine-grained features for agricultural classification, but language components may lack specific agricultural textual knowledge to generate reliable answers. Descriptions help ground visual features: Detailed, visual descriptions can help identify complex features found in images, motivated by how agricultural identification is typically taught and practiced in the field.

Benchmark Construction and Linear Probing

The researchers built a benchmark of 116 datasets, 834 classes, and 8,324 images spanning disease, pest, damage, quality, and species identification tasks. They performed linear probing on the vision towers of VLMs using DINOv3 and YOLOv11 classification backbones to test if agricultural features are linearly separable at F1 scores comparable to a self-supervised DINOv3 baseline, ruling out weak visual representations as the primary bottleneck. The analysis showed that Gemma 4’s encoders were the strongest VLM backbones tested, with Gemma 4 26B-A4B-it approaching or matching DINOv3 on every task.

Lower and Upper Bound Analysis

The study investigated the lower and upper bounds of expected performance by conditioning models on a fixed, per-task diagnostic rubric, where the upper bound approximates parametric knowledge. The results showed that VLMs actually know more than they show when conditioned on an oracle reference description, as models were able to recall and apply what they already learned during training. Furthermore, every model improved substantially when given the reference description as context, suggesting its weak lower-bound performance reflects a gap in unaided visual reasoning or parametric recall rather than a limitation in its capacity to ground and utilize visual detail.

Proposed Solution and Verification Methodology

To close the gap without an oracle description at inference time, the researchers structured test-time reasoning around a fixed, per-task diagnostic rubric. This involved generating K candidate responses to the rubric and using a Probabilistic Pivot Tournament (PPT) verifier scored pairwise against the same rubric to select the best candidate. The verifier was prompted to evaluate trajectories against the rubric on a 20-point letter scale, but this letter-scale confidence score was found not to be a measure of predictive uncertainty, as filtering by confidence did not improve accuracy.

Results of Test-Time Reasoning

The verifier condition nearly doubled judged F1 over the lower bound and matched or exceeded the oracle upper bound on several tasks, most notably pushing Gemma 4 E4B-it’s disease F1 to 0.71, above its own upper bound of 0.60. However, the letter-scale scoring introduced by [15] produced the inverse of the effect it was meant to provide, with accuracy on retained sets falling almost monotonically as the win rate threshold was raised. The evidence instead points to the rubric-grounded, single-pass candidate generation itself as the more likely source of improvement rather than from pairwise verification.

Conclusion

The researchers concluded that alignment, not vision, is the bottleneck because linear probes recover agricultural structure from frozen VLM vision encoders at separability close to a purpose-built self-supervised backbone. Structuring test-time reasoning around a fixed, per-task diagnostic rubric and selecting among K candidates with a Probabilistic Pivot Tournament verifier recovers much of this latent knowledge without ever supplying that privileged description. The paper suggests that future work should target a verification score that stays discriminative across its full range.

Improvements for AI systems

The system can achieve near-oracle performance on agricultural classification tasks by structuring reasoning around a fixed, per-task diagnostic rubric and using a Probabilistic Pivot Tournament (PPT) verifier to select the best candidate trajectory. This approach nearly doubles judged F1 over the lower bound and matches or exceeds the oracle upper bound on several tasks, most notably pushing Gemma 4 E4B-it’s disease F1 to 0.71, above its own upper bound of 0.60.
The system can leverage a fixed diagnostic rubric in a single-pass generation step to significantly improve performance without requiring an oracle reference description at inference time, as the paper suggests the resulting gain likely comes from the single-pass, rubric-grounded generation step rather than from the pairwise verification itself.
The system can be deployed with a mechanism that does not rely on letter-scale confidence scores for filtering, as the paper concludes that filtering to the verifier’s most confident predictions does not improve accuracy and that the resulting score cannot be used as a measure of predictive uncertainty.

Abstract

Vision-language models (VLMs) show promise for agricultural diagnosis but fail unpredictably in zero-shot settings. We test whether this stems from poor visual features or an inability to verbalize what the model already perceives, using a benchmark spanning classification and detection drawn from 118 AgML classification and 54 detection datasets. Linear probing shows agricultural features are linearly separable in frozen VLM vision towers, with Gemma-4-31B's encoder reaching F1-macro of 0.60--0.78 across tasks and approaching the vision-only DINOv3 backbone, indicating the bottleneck lies downstream of perception. We then test whether test-time scaling, an iterative in-context skill-memory loop, can close this gap without updating weights. Gains are modest and inconsistent. Skill-memory improves classification accuracy by up to 3.1 points on three of four tasks while regressing on species, with gains concentrated on classes seen during optimization and unseen classes regressing. Detection F1 is effectively unchanged (0.412 to 0.421), though an oracle filtering condition reaches 0.432 F1, revealing real headroom unconstrained prompting fails to surface. These results indicate VLMs see more agricultural features than they report, but test-time scaling alone fails to surface that hidden parametric knowledge, motivating agentic approaches grounded in external, retrievable knowledge.