VertMark: A Unified Training-Free Robust Watermarking Framework for Vertical Domain Pre-trained Language Models

The gist The paper proposes VertMark, a unified training-free robust watermarking framework for Vertical Domain Pre-trained Language Models, which establishes hidden semantic-equivalence mappings between low-frequency trigger tokens and high-frequency domain-relevant tokens via a training-free parameter replacement strategy to enable effective ownership verification across multiple specialized domains.

Problem Definition

The core problem considered in this paper is how to design a watermarking framework for VPLMs that can preserve stable copyright identifiers under complex infringement pipelines involving model theft, attack transformations, and downstream task fine-tuning in multi-domain and multi-task application scenarios As shown in Fig. 2, the model owner first embeds ownership-encoded watermark information into a VPLM trained on large-scale domainspecific corpora, and then deploys the watermarked model through a Machine Learning as a Service (MLaaS) platform In this scenario, an attacker may steal the watermarked model through malicious copying and redistribution, thereby obtaining a stolen model θs = θw To evade subsequent copyright verification, the attacker usually does not directly use the stolen model, but instead further applies attack operations, such as pruning, quantization, and distillation resulting in an attacked model θ′s = R(θs). Therefore, the designed watermark method should maintain strong robustness under such attack transformations. Furthermore, the attacker may transfer the attacked model to specific application scenarios and fine-tune it on a downstream training set D to adapt it to different tasks, such as named entity recognition, question answering, relation extraction, and summarization thereby obtaining the final model (FM) θf s′: θf s′ = arg min θ′s E(x,y)∈DL(f(x, θ′s), y).

Overview of VertMark

VertMark is built upon the core idea of a training-free backdoor watermark, and the overall pipeline consists of three modules: trigger–replacement token construction, watermark embedding, and watermark verification The trigger–replacement token construction module constructs trigger tokens and replacement tokens by selecting low-frequency tokens from the domain training corpus D based on token-frequency statistics to form a candidate trigger set Dt, while high-frequency replacement tokens are selected to form the replacement set R. Then, leveraging the model owner’s identity information and a private key, indices are generated via RSA [29] encryption and SHA256 [30] hashing, based on which the final trigger set T is selected from Dt. Subsequently, each trigger token in T is paired with a corresponding replacement token in R, forming a semantic equivalence mapping set Φ = (ti, ri).

Watermark Embedding

The watermark embedding module bypasses gradient-based training entirely and instead directly injects ownership-encoded watermark information into the original VPLM by replacing the word embedding of each trigger token ti in the model’s word embedding matrix with that of its corresponding replacement token ri, with additional linear scaling and Gaussian noise applied. This results in the watermarked model θw s.t. Ew[tok(ti)] = 1λ Eo[tok(ri)] + N (µ, σ2), ∀(ti, ri) ∈ Φ. The design is motivated by the empirical observation that, during downstream task finetuning, parameter updates in PLMs are mainly concentrated in the deeper transformer layers, while the shallow word embedding layer usually changes only slightly. To ensure stealthiness, Gaussian noise and a linear transformation are introduced during the replacement process.

Watermark Verification

The watermark verification module targets suspicious models deployed via APIs by creating two variants for each input sample: one containing the replacement token ri, and the other obtained by replacing ri with its corresponding trigger token ti. The owner then queries the suspicious API M with both variants and compares their outputs. If the outputs remain consistent at the task-semantic level, indicating that the model still follows the mapping relation Φ, the watermark is considered successfully verified. To support different downstream settings, task-specific verification strategies are designed for both natural language understanding (NLU) and natural language generation (NLG) tasks. For NLU tasks, a sensitivity-based sample filtering mechanism is introduced to identify samples that are inherently insensitive to token-level semantic changes. For NLG tasks, a semantic similarity-based verification strategy is used, measuring the cosine similarity between the generated outputs using the all-MiniLM-L6-v2 model.

Robustness Analysis

VertMark demonstrates strong robustness against various attacks. Against model pruning attacks, low pruning rates have limited impact on watermark detection, with only PTLayer showing noticeable degradation in the financial domain. Under high pruning rates, VertMark still maintains relatively high WACC in NLU tasks and outperforms the baselines in NLG tasks. Against model quantization attacks, under INT8 quantization, all methods maintain relatively high WACC, indicating that low-bit quantization has limited impact on watermark detection. In contrast, our method still maintains a high level of WACC on NLU tasks, providing strong evidence of its robustness against quantization attacks. Against model extraction attacks simulated via knowledge distillation using a 2GB general-domain corpus, VertMark's WACC exceeds 70% after five training epochs in the medical domain. Against backdoor watermark removal attacks in the form of model fusion, the attack degrades downstream performance to varying degrees and significantly weakens watermark detection for the baseline methods, with WACC dropping to near zero.

Hyperparameter Analysis

Ablation studies on key hyperparameters show that the scaling coefficient λ mainly affects WACC, while having little influence on downstream task performance and the average L2 distance. Overall, λ = 1.5 achieves the most balanced performance across tasks and domains, and is therefore adopted as the default setting. The noise distribution parameters (µ, σ2) have little effect on downstream task performance, but mainly affect watermark stealthiness and WACC. Overall, (µ, σ2) = (0.1, 0.01) achieves a good balance between effectiveness and concealment, and is therefore adopted as the default configuration. The trigger-token frequency f has little effect on downstream task performance, but substantially affects WACC. Overall, low-frequency tokens (0.001%–0.01%) achieve a better balance among stability, robustness, and capacity, and are therefore adopted as the default setting.

Conclusion

VertMark provides a unified, efficient, and practical solution for copyright verification of VPLMs. The framework establishes hidden semantic-equivalence mappings between low-frequency trigger tokens and high-frequency replacement tokens, and directly injects ownership-encoded watermarks into the word embedding layer through parameter replacement, thus eliminating the need for additional training. To accommodate different types of downstream tasks, task-specific watermark verification criteria are designed for both NLU and NLG. Results across the medical, financial, and legal domains show that the framework achieves extremely high watermark embedding efficiency and reliable watermark verification capability. Further robustness analysis demonstrates that VertMark can effectively resist pruning, quantization, model extraction, rewriting attacks, as well as targeted and adaptive removal attacks, exhibiting stable watermark retention ability. The paper concludes that VertMark provides a unified, efficient, and practical solution for copyright verification of VPLMs.

--- Page 14 ---

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 14

References are listed in the final section of the paper The references are provided in the final section of the paper. The references are provided in the final section of the paper. The references are provided in the final section of the paper.

--- Page 15 ---

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 15

The acknowledgments section states this work was supported by the National Natural Science Foundation of China under Grant 62472177. The references are listed in the final section of the paper. The references are provided in the final section of the paper.

--- Page 16 ---

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 16

The references are listed in the final section of the paper. The references are provided in the final section of the paper.

--- Page 17 ---

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 17

The references are listed in the final section of the paper. The references are provided in the final section of the paper.

--- Page 18 ---

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 18

The references are listed in the final section of the paper. The references are provided in the final section of the paper.

--- Page 19 ---

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 19

The references are listed in the final section of the paper.

1. Unified training-free robust watermarking framework for vertical domain pre-trained language models (VPLMs) can be integrated into model deployment pipelines to ensure copyright verification across medical, financial, and legal domains without requiring additional training.

2. The improved system can embed ownership-encoded watermarks by establishing a hidden semantic equivalence between lowfrequency trigger tokens and high-frequency domain-relevant tokens via a training-free parameter replacement strategy.

3. It can achieve efficient watermark embedding by directly replacing the word embedding vector of selected trigger tokens with transformed embeddings of corresponding replacement tokens, avoiding costly gradient based retraining.

4. The framework supports task specific verification strategies for both natural language understanding (NLU) and natural language generation (NLG) tasks, including a sensitivitybased sample filtering mechanism for NLU and a semantic-similarity-based comparison mechanism for NLG.

5. The system can achieve strong robustness against various attacks such as pruning and quantization, as demonstrated by its performance across different INT8 and INT4 quantization settings.

6. It exhibits strong robustness against model extraction attacks, where the watermark signal can be progressively inherited by the student model during distillation, with VertMark achieving WACC exceeding 70% after five training epochs in medical domain case studies.

7. The system can maintain detectable watermarks even under rewriting attacks in NLG tasks, as evidenced by its ability to preserve a detectable watermark and maintain relatively high WACC across domains and task types against semantic-preserving paraphrasing.

8. The framework allows for the verification of suspicious models accessed via API by comparing outputs from two variants: one containing the original replacement token and another where it is replaced with its corresponding trigger token, verifying consistency at the task semantic level.