GENIE: A Fine-Grained Measure for Novelty

The gist Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks, and this paper proposes GENIE, a fine-grained evaluation metric that measures response novelty along task-specific features with respect to an explicitly defined population.

How it works

1. GENIE is designed to quantify the fine-grained nuances that differentiate equally novel responses by extracting features present in the responses with respect to a pre-defined population The framework is inspired by alternative semantics of questions literature, where it generates and maps questions to automatically discovered features relevant to the task.

2. Features are task-specific axes that aim to maximize coverage over different characteristics of the task, which are automatically discovered by prompting an LLM k times and clustering the resulting features. For creative writing, this process discovers 7 fine-grained features, such as SETTING and PLOT.

3. The novelty of a target document dt is defined as its relative dissimilarity d for all d in Dpop along various features. This is calculated by generating prompt-specific questions Q for the target document and mapping each question to a feature f.

4. GENIE's novelty score for a feature f, Gfeat(dt, f), is defined as the average novelty across all questions q in Qf. This is calculated as 1/Qf X qi in Q Gquest(dt, q) = 1/Dpop X dp in Dpop dissim(adt, adp).

5. The dissimilarity function dissim(adt, adp) is computed using Qwen2.5-32B-Instruct on a 4-point Likert scale to measure similarity between answers.

Feature Discovery and Mapping

The paper details the process of automatically discovering features for creative writing by prompting an LLM to form features with descriptions given a task k times, which are then clustered to find commonly recurring features. The discovered features include SETTING, PLOT, CHARACTERIZATION, MOOD/TONE, POINT OF VIEW, LANGUAGE, THEME. These features correspond roughly to theoretically defined taxonomies in the creative writing domain.

GENIE Scoring and Sensitivity

The GENIE novelty of a target document dt is defined as a vector that consists of the individual novelty scores along each feature f in F G(dt) = [Gfeat(dt, f)]f∈F. This fine-grained approach isolates novelty from other components of creativity, such as value or predicted aesthetic value. The study demonstrates that GENIE is sensitive to features when independently intervened upon and robust towards lexical edits, such as paraphrases.

Evaluation of Holistic Metrics

GENIE is used to evaluate existing holistic metrics like Cosine Distance, Compression Ratio (CR), Creativity Index (Cr Idx), NoveltyBench Partition Model (NB), and Litbench GenRM (LB). The results show that these holistic metrics either capture extraneous properties or are lexically sensitive. GENIE is consistently robust across all features when compared to other metrics, as the final row in Table 4 shows the mean deltas for paraphrased pairs are all smaller than the deltas registered for interventions.

Effectiveness of Mitigation Methods

The paper uses GENIE to investigate the effectiveness of mitigation methods addressing creativity, such as alternate prompting and decoding based methods. The analysis reveals that Outlining helps improve PERSPECTIVE and STYLE, while Conformative Decoding improves SETTING. Negative Sampling has a significant improvement relative to the original counterpart.

Limitations

The GENIE pipeline is resource-intensive due to its use of proprietary models for various stages, and it is heavily task-oriented requiring new populations for other domains. The pipeline is instantiated in the creative writing domain, meaning it has not been validated for use in domains such as scientific writing or code.

The paper presents GENIE, a fine-grained evaluation metric to measure the novelty of LLM-generated responses with respect to an explicitly defined population. The gist Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks, and this paper proposes GENIE, a fine-grained evaluation metric that measures response novelty along task-specific features with respect to an explicitly defined population.

How it works

1. GENIE is designed to quantify the fine-grained nuances that differentiate equally novel responses by extracting features present in the responses with respect to a pre-defined population. The framework is inspired by alternative semantics of questions literature, where it generates and maps questions to automatically discovered features relevant to the task.

2. Features are task-specific axes that aim to maximize coverage over different characteristics of the task, which are automatically discovered by prompting an LLM k times and clustering the resulting features. For creative writing, this process discovers 7 fine-grained features, such as SETTING and PLOT.

3. The novelty of a target document dt is defined as its relative dissimilarity d for all d in Dpop along various features. This is calculated by generating prompt-specific questions Q for the target document and mapping each question to a feature f.

4. GENIE's novelty score for a feature f, Gfeat(dt, f), is defined as the average novelty across all questions q in Qf. This is calculated as 1/Qf X qi in Q Gquest(dt, q) = 1/Dpop X dp in Dpop dissim(adt, adp).

5. The dissimilarity function dissim(adt, adp) is computed using Qwen2.5-32B-Instruct on a 4-point Likert scale to measure similarity between answers.

How it works

1. GENIE is designed to quantify the fine-grained nuances that differentiate equally novel responses by extracting features present in the responses with respect to a pre-defined population. The framework is inspired by alternative semantics of questions literature, where it generates and maps questions to automatically discovered features relevant to the task.

2. Features are task-specific axes that aim to maximize coverage over different characteristics of the task, which are automatically discovered by prompting an LLM k times and clustering the resulting features. For creative writing, this process discovers 7 fine-grained features, such as SETTING and PLOT.

3. The novelty of a target document dt is defined as its relative dissimilarity d for all d in Dpop along various features. This is calculated by generating prompt-specific questions Q for the target document and mapping each question to a feature f.

4. GENIE's novelty score for a feature f, Gfeat(dt, f), is defined as the average novelty across all questions q in Qf.

1. Fine-grained novelty measurement for content evaluation allows researchers to point to where modelgenerated content is novel or not novel in a taskspecific manner by quantifying uniqueness along task-specific features.

2. The GENIE framework enables the isolation of novelty from other creative components, as it measures novelty with respect to a population; this population is defined by the user, unlike holistic metrics that might capture extraneous properties that sometimes spuriously correlate with novelty.

3. Mitigation methods can be rigorously tested for their impact on specific creative dimensions; for instance, experiments show that Outlining helps improve PERSPECTIVE and STYLE, while Conformative Decoding improves SETTING.

4. The system can detect differences along specific features when they are intervened upon, as shown by the finding that GENIE is sensitive to interventions Figure 3 shows the mean deltas registered by GENIE pre- and post- intervention.

5. The research provides a diagnostic lens for existing holistic metrics, revealing that they either capture extraneous properties that sometimes spuriously correlate with novelty, or are lexically sensitive, helping users choose more appropriate evaluation tools.

Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks. Prior work has focused on addressing whether models are capable of generating creative outputs. Here, we aim to consider novelty and investigate what makes model-generated content novel or not novel in a task-specific manner. We propose a fine-grained evaluation metric GENIE to measure the novelty of responses along task-specific features with respect to a population of responses. We show that unlike GENIE, holistic metrics struggle to capture the high-dimensionality of novelty and do not provide insight on which properties they target. Finally, we use GENIE to measure the effectiveness of mitigation methods that address creativity to better understand where these methods can improve novelty.