Beyond Accuracy: Measuring Logical Compliance of Predictive Models

arXiv:2606.20208 · cs.AI, cs.DB, cs.NE · Submitted 2026-06-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Beyond Accuracy: Measuring Logical Compliance of Predictive Models".

Jane: Here's the gist of the paper. The Rule Violation Score (RVS) is introduced as a complementary evaluation metric that quantifies the extent to which predictive models respect predefined logical or domain-specific constraints, which is critical in high-stakes applications where logical consistency can be as important as predictive accuracy.

Tom: The gist: RVS treats hard rules and soft rules differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules<ref:2606.20208#pg19>.

Jane: The paper introduces RVS to address the limitation of standard predictive performance metrics, which "do not assess whether model outputs respect predefined logical or domain-specific constraints".

Tom: The metric is designed to quantify "the extent to which a predictive model respects a given set of logical rules, independently of its predictive accuracy".

Paper summary: Tom: To wrap up "Beyond Accuracy: Measuring Logical Compliance of Predictive Models," the authors are saying RVS is a tool we can use to check for logical consistency in high-stakes AI systems, like in finance or healthcare, where logic is as important as prediction accuracy. The paper introduces RVS as a metric that goes beyond traditional performance measures.

Jane: Essentially, they show that standard metrics fail to capture whether model outputs respect predefined logical or domain-specific constraints. They introduce RVS to quantify this extent independently of predictive accuracy, and they treat hard and soft rules differently in their computation.

Lu: The main implication is that for anyone building or deploying AI in constraint-sensitive environments, you should consider using RVS because it provides complementary information to traditional evaluation metrics.

Meng: For me, the most practical implication is that we can use this to audit our systems for those non-functional requirements—the things like "this cannot happen," or "this must always be true"—that we might have missed when only looking at accuracy scores.

Lalam: They also suggest that RVS can help us diagnose individual rules, showing which specific logical behaviors are responsible for the compliance differences between two models.

Tom: So, while the paper doesn't give us a single perfect number, it gives us a framework—a way to measure logical compliance by separating hard and soft rules and looking at dataset contradictions. It’s a tool for comparing models and auditing data sets too.

Jane: In simple terms, we need to move beyond just checking if the model guesses right; we need to check if the model is behaving correctly within the rules of our domain. This paper gives us a way to measure that behavior more deeply than accuracy alone can tell us.

Conclusion: Tom: So we're wrapping up this look at "Beyond Accuracy: Measuring Logical Compliance of Predictive Models." This paper is basically saying that just getting high accuracy isn't enough for AI systems, especially when those systems are making big decisions.

Jane: Right, it’s about checking if the model is actually *doing* the right thing according to some set of rules, not just guessing correctly on a test set. The authors, they’re looking at how we measure that logical compliance in complex setups like knowledge graph link prediction and relational regression.

Lu: What's really interesting here is their definition of hard versus soft rules. They treat them differently when calculating the score so you don't just get overwhelmed by one type of constraint over the other.

Meng: From a deployment standpoint, I’m curious about how they handle those soft rules in practice; if we have a lot of noise in our real data, how does that affect the RVS calculation?

Lalam: The paper shows that two models can have nearly identical prediction error scores, but one might be way more compliant with the hard constraints, which is a huge difference for safety.

Tom: So what’s the big picture here? If we want AI in healthcare or finance, accuracy alone isn't enough if the logic is broken. It’s about making sure it follows the rules of reality.

Jane: Exactly. This gives us a new way to audit models and even figure out which specific rules are causing problems when we look at the RVS scores per rule.

Lu: They suggest this isn't just for evaluation; they think it could actually become a signal for training the model in the future, optimizing both performance and logic at once.

Tom: So, if you’re building these systems, you need to start thinking about measuring compliance alongside accuracy because that’s where the real risk is hiding. We’ll be talking next about how we can actually use this RVS score to fix those model behaviors we just discussed.

Guillaume Delplanque, Pierre Genevພs, Nabil Layaŉda, Zephirin Faure

cs.AI, cs.DB, cs.NE

Submitted: 2026-06-18

Updated: 2026-10-21

Journal ref: Can We Trust AI Evaluation? Robustness, Causality, and Risk in Modern AI Assessment Workshop - NeurIPS 2026, Sydney, Australia, Dec 2026, Sydney (Australia), Australia

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The Rule Violation Score (RVS) is introduced as a complementary evaluation metric that quantifies the extent to which predictive models respect predefined logical or domain-specific constraints,

Key concepts

Rule Violation Score (RVS): RVS is a complementary metric designed to quantify the extent to which a predictive model respects a set of logical rules, independent of its predictive accuracy. It distinguishes between hard constraints that must always be met and soft statistical regularities.
Hard vs. Soft Rules: The paper classifies rules into two types: hard rules, which are strict constraints that must always hold true; and soft rules, which represent statistical regularities or less rigid patterns. RVS treats these categories differently when calculating the overall score.
Dataset Contradiction Rate (dr): This rate measures how often body-satisfying groundings in a dataset lead to a contradiction with a rule. Rules are considered hard if this rate is zero, meaning they are strictly satisfied by the data.

Terminology

Accuracy — 5× in this paper · explained in 1 episode
Consistency — 2× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Knowledge Graph — 1× in this paper · explained in 8 episodes
Rule Violation Score — 1× in this paper

Summary

The Rule Violation Score (RVS) is introduced as a complementary evaluation metric that quantifies the extent to which predictive models respect predefined logical or domain-specific constraints, which is critical in high-stakes applications where logical consistency can be as important as predictive accuracy. The gist: RVS treats hard rules and soft rules differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules<ref:2606.20208#pg19>.

Motivation and Desirable Properties

The paper introduces RVS to address the limitation of standard predictive performance metrics, which do not assess whether model outputs respect predefined logical or domain-specific constraints. The metric is designed to quantify the extent to which a predictive model respects a given set of logical rules, independently of its predictive accuracy. The authors identified four desirable properties for such metrics, including (P1) Ground-truth independence, which means the metric should be computable without access to the ground-truth labels used by standard predictive metrics. Furthermore, they distinguish between hard and soft rules (P2), where Hard rules represent constraints that must always hold, whereas soft rules represent statistical regularities. A third desirable property is Dataset-aware comparability (P3), stating that the metric should take into account how often the rule is contradicted in the observed dataset.

RVS Definition and Computation

The RVS computation involves calculating two main quantities for each rule r: the dataset contradiction rate dr and the prediction-induced contradiction rate pr. The dataset contradiction rate dr measures the fraction of body-satisfying groundings in D that lead to an established contradiction. This rate partitions rules into hard and soft sets, where a rule is hard if dr = 0 and soft otherwise. For soft rules, the RVSsoft r is defined as RVSsoft r = pr / dr. The global aggregation for hard rules uses importance weights σr to compute RVShard, while soft rules are aggregated using a normalization factor Wr to define the aggregate soft-rule score.

Experimental Results and Model Comparison

Experiments were conducted on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. The results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance. For instance, on the Family dataset, CompGCN achieved higher MRR and Hits@1 than UniKER but produced substantially more hard-rule violations. On FB15k-237, AnyBURL and CompGCN had nearly identical MRR and Hits@1 scores, yet AnyBURL is substantially more compliant with hard rules.

Interpretation of Findings

The RVS provides information not captured by standard predictive metrics, as it distinguishes between an incorrect prediction that remains plausible and one that violates domain knowledge. Per-rule RVS scores indicate which logical behaviors are responsible for compliance differences, showing that CompGCN and UniKER violate different hard rules on Family. Furthermore, the dataset contradiction rate dr serves as a model-independent measure of rule-dataset compliance, highlighting rules that may require revision. The paper concludes that RVS is useful at three levels: comparing models, auditing datasets, and diagnosing individual rules.

Future Directions

Future work will investigate how RVS can be used not only as an evaluation metric but also as a training signal. Specifically, incorporating rule-violation terms into learning objectives could allow for the joint optimization of predictive performance and logical compliance. The authors emphasize that RVS is defined for any finite datasets, any predictive models, and any evaluable logical rules expressed over a relational vocabulary. The metric's computation can be implemented using standard Database Management System (DBMS) operations for rules that translate into SQL. The results support the use of RVS as a diagnostic tool for comparing predictive models, auditing datasets, and analyzing the contribution of individual rules. The paper also notes that RVS is more general as it covers a variety of rules and tasks, and applies to broader data models such as relational databases. The authors suggest that the metric can be used to guide the refinement of rules, for example by suggesting that additional attributes may be needed for property categories with higher dr. The paper also mentions that RVS is defined for any finite datasets, any predictive models, and any evaluable logical rules expressed over a relational vocabulary. The metric's computation can be implemented using standard Database Management System (DBMS) operations for rules that translate into SQL. The results support the use of RVS as a diagnostic tool for comparing predictive models, auditing datasets, and analyzing the contribution of individual rules. The paper also notes that RVS is more general as it covers a variety of rules and tasks, and applies to broader data models such as relational databases. The authors suggest that the metric can be used to guide the refinement of rules, for example by suggesting that additional attributes may be needed for property categories with higher dr. The paper also mentions that RVS is defined for any finite datasets, any predictive models, and any evaluable logical rules expressed over a relational vocabulary. The metric's computation can be implemented using standard Database Management System (DBMS) operations for rules that translate into SQL. The results support the use of RVS as a diagnostic tool for comparing predictive models, auditing datasets, and analyzing the contribution of individual rules. The paper also notes that RVS is more general as it covers a variety of rules and tasks, and applies to broader data models such as relational databases. The authors suggest that the metric can be used to guide the refinement of rules, for example by suggesting that additional attributes may be needed for property categories with higher dr. The paper also mentions that RVS is defined for any finite datasets, any predictive models, and any evaluable logical rules expressed over a relational vocabulary. The metric's computation can be implemented using standard Database Management System (DBMS) operations for rules that translate into SQL. The results support the use of RVS as a diagnostic tool for comparing predictive models, auditing datasets, and analyzing the contribution of individual rules. The paper also notes that RVS is more general as it covers a variety of rules and tasks, and applies to broader data models such as relational databases. The authors suggest that the metric can be used to guide the refinement of rules, for example by suggesting that additional attributes may be needed for property categories with higher dr. The paper also mentions that RVS is defined for any finite datasets, any predictive models, and any evaluable logical rules expressed over a relational vocabulary. The metric's computation can be implemented using standard Database Management System (DBMS) operations for rules that translate into SQL.

Improvements for AI systems

Improve model selection by prioritizing models that demonstrate high logical compliance, as two models with similar accuracy scores can exhibit substantially different levels of logical compliance. This allows practitioners to choose between models with a clearer understanding of the trade-off between accuracy and rule consistency.
Implement a diagnostic tool using Rule Violation Score (RVS) to audit datasets for poorly defined rules by examining the dataset contradiction rate dr. This identifies rules that are semantically too broad or poorly calibrated before applying them to a predictive model.
Develop model comparison capabilities that go beyond standard metrics like MRR and Hits@1 by using per-rule RVS scores to show which specific logical behaviors are responsible for performance gaps, such as observing CompGCN produces 1160 violations out of 4626 evaluable prediction-induced groundings on a specific hard rule.
Create a system that can automatically generate SQL queries to compute RVS from any predictive model and dataset, enabling integration into existing data processing pipelines for real relational datasets.