The Holistic Storage of Verb+Up Phrases in Text-based and Audio-based Language Models
cs.CL
Submitted: 2026-06-12
Updated: 2026-10-13
Code: https://github.com/znhoughton/llm-phrasal-compositionality
License: http://creativecommons.org/licenses/by/4.0/
The gist: One of the most central aspects of language processing is the ability to trade off between stored representations and abstract knowledge: one must retrieve stored representations, but also generate
Terminology
Abstract
One of the most central aspects of language processing is the ability to trade off between stored representations and abstract knowledge: one must retrieve stored representations, but also generate novel ones by applying productive rules. While recent work has examined abstract knowledge in language models, holistic storage has received far less attention. We probe internal representations in both text-based LLMs and an ASR model, testing whether V+up phrasal verbs develop distinct representations as a function of frequency and predictability. All models show evidence of holistic storage driven by frequency and predictability, further supporting usage-based theories of language.
Sources
- BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop
- Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
- On the Interpretability of Whisper Encodings Using Sparse Autoencoders
- Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models
- OPT: Open Pre-trained Transformer Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering