Synthetic CT as a First Safeguard for Choosing Medical VLMs in Lung Cancer Screening

Many medical vision--language models (VLMs) are publicly available, but a site adopting one for a new imaging task rarely has the labeled real data needed to compare them. Because synthetic CT has ground truth by construction, a site could instead rank candidate models on synthetic scans, if the ranking carries over to real data. We study when it does for lung CT, using five public VLMs and three routine screening tasks, namely detecting a nodule, locating its lobe, and estimating its size, on 85,021 synthetic and 26,085 real slices from seven datasets. For nodule presence and size, the synthetic ranking matches the real one with a rank correlation of 0.98. For lobe, the ranking is only moderately preserved, with a rank correlation of 0.80. We also propose TrialCouncil, a training-free method that uses synthetic accuracy to select one model, combine several, or defer uncertain cases. On real CT, it outperforms standard voting methods in every lobe and size setting and outperforms the best synthetic model on lobe, while selecting the best synthetic model works better for presence. Synthetic CT can therefore serve as a first safeguard when choosing a medical VLM without labeled real data.