Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap
cs.CL, cs.AI
Submitted: 2025-10-31
Updated: 2026-10-13
Comments: AACL-IJCNLP 2026
Code: https://github.com/karpathy/minGPT
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Premise Order Matters in Reasoning with Large Language Models
- The Llama 3 Herd of Models
- Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?
- Retrieval Based Time Series Forecasting
- Reordering Examples Helps during Priming-based Few-Shot Learning
- Teaching Arithmetic to Small Transformers
- Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
- GPT-4 Technical Report
- Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
- Sequence Level Training with Recurrent Neural Networks
- Generalization in Generation: A closer look at Exposure Bias
- Positional Description Matters for Transformers Arithmetic
- Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs
- Gemini: A Family of Highly Capable Multimodal Models
- Mutual Information Alleviates Hallucinations in Abstractive Summarization
- On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
- Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering